Vehicular monitoring systems using image processing

ABSTRACT

Vehicular monitoring arrangement for monitoring an environment of the vehicle including at least one active pixel camera for obtaining images of the environment of the vehicle and a processor coupled to the active pixel camera(s) for determining at least one characteristic of an object in the environment based on the images obtained by the active pixel camera(s). The active pixel camera can be arranged in a headliner, roof or ceiling of the vehicle to obtain images of an interior environment of the vehicle, in an A-pillar or B-pillar of the vehicle to obtain images of an interior environment of the vehicle, or in a roof, ceiling, B-pillar or C-pillar of the vehicle to obtain images of an interior environment of the vehicle behind a front seat of the vehicle. The determined characteristic can be used to enable optimal control of a reactive component, system or subsystem coupled to the processor. When the reactive component is an airbag assembly including at least one airbag, the processor can be designed to control at least one deployment parameter of the airbag(s).

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation-in-part of U.S. patentapplication Ser. No. 10/116,808 filed Apr. 5, 2002 which is:

[0002] 1) a continuation-in-part of U.S. patent application Ser. No.09/925,043 filed Aug. 8, 2001 which is:

[0003] a) a continuation-in-part of U.S. patent application Ser. No.09/765,559 filed Jan. 19, 2001; and

[0004] b) a continuation-in-part of U.S. patent application Ser. No.09/389,947 filed Sep. 3, 1999, now U.S. Pat. No. 6,393,133; and

[0005] 2) a continuation-in-part of U.S. patent application Ser. No.09/838,919 filed Apr. 20, 2001, now U.S. Pat. No. 6,442,465, which is:

[0006] a) a continuation-in-part of U.S. patent application Ser. No.09/765,559 filed Jan. 19, 2001 which is a continuation-in-part of U.S.patent application Ser. No. 09/476,255 filed Dec. 30, 1999, now U.S.Pat. No. 6,324,453, which claims priority under 35 U.S.C. §119(e) ofU.S. provisional patent application Ser. No. 60/114,507 filed Dec. 31,1998; and

[0007] b) a continuation-in-part of U.S. patent application Ser. No.09/389,947 filed Sep. 3, 1999, now U.S. Pat. No. 6,393,133, which is acontinuation-in-part of U.S. patent application Ser. No. 09/200,614,filed Nov. 30, 1998, now U.S. Pat. No. 6,141,432, which is acontinuation of U.S. patent application Ser. No. 08/474,786 filed Jun.7, 1995, now U.S. Pat. No. 5,845,000, all of which are incorporated byreference herein.

[0008] This application claims priority under 35 U.S.C. §119(e) of U.S.provisional patent application Ser. No. 60/114,507 filed Dec. 31, 1998through the parent applications.

FIELD OF THE INVENTION

[0009] The present invention relates to apparatus and methods formonitoring environments in and outside of a vehicle using imageprocessing.

[0010] The present invention also relates to arrangements for detectingthe presence, type and/or position of occupants in vehicles and objectsexterior of vehicles, e.g., in a driver's blind spot, primarily usingoptics.

[0011] The present invention also relates to apparatus and methods fordetermining a distance between objects in an environment in and outsideof a vehicle by image processing techniques.

BACKGROUND OF THE INVENTION

[0012] 1. Prior Art on Out of Position Occupants and Rear Facing ChildSeats

[0013] Whereas thousands of lives have been saved by airbags, a largenumber of people have also been injured, some seriously, by thedeploying airbag, and over 100 people have now been killed. Thus,significant improvements need to be made to airbag systems. As discussedin detail in U.S. Pat. No. 5,653,462 referenced above, for a variety ofreasons vehicle occupants may be too close to the airbag before itdeploys and can be seriously injured or killed as a result of thedeployment thereof. Also, a child in a rear facing child seat that isplaced on the right front passenger seat is in danger of being seriouslyinjured if the passenger airbag deploys. For these reasons and, as firstpublicly disclosed in Breed, D. S. “How Airbags Work” presented at theInternational Conference on Seatbelts and Airbags in 1993, in Canada,occupant position sensing and rear facing child seat detection systemsare required.

[0014] Initially, these systems will solve the out-of-position occupantand the rear facing child seat problems related to current airbagsystems and prevent unneeded airbag deployments when a front seat isunoccupied. However, airbags are now under development to protect rearseat occupants in vehicle crashes and all occupants in side impacts. Asystem will therefore be needed to detect the presence of occupants,determine if they are out-of-position and to identify the presence of arear facing child seat in the rear seat. Future automobiles are expectedto have eight or more airbags as protection is sought for rear seatoccupants and from side impacts. In addition to eliminating thedisturbance and possible harm of unnecessary airbag deployments, thecost of replacing these airbags will be excessive if they all deploy inan accident needlessly.

[0015] Inflators now exist which will adjust the amount of gas flowingto the airbag to account for the size and position of the occupant andfor the severity of the accident. The vehicle identification andmonitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782 willcontrol such inflators based on the presence and position of vehicleoccupants or of a rear facing child seat. As discussed more fully below,the instant invention is an improvement on that VIMS system and uses anadvanced optical system comprising one or more CCD (charge coupleddevice) or CMOS arrays and particularly active pixel arrays plus asource of illumination preferably combined with a trained neural networkpattern recognition system.

[0016] Others have observed the need for an occupant out-of-positionsensor and several methods have been disclosed in U.S. patents fordetermining the position of an occupant of a motor vehicle. Each ofthese systems, however, has significant limitations. For example, inWhite et al. (U.S. Pat. No. 5,071,160), a single acoustic sensor anddetector is described and, as illustrated, is mounted lower than thesteering wheel. White et al. correctly perceive that such a sensor couldbe defeated, and the airbag falsely deployed, by an occupant adjustingthe control knobs on the radio and thus they suggest the use of aplurality of such sensors.

[0017] Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety ofmethods of measuring the change in position of an occupant includingultrasonic, active or passive infrared and microwave radar sensors, andan electric eye. The sensors measure the change in position of anoccupant during a crash and use that information to access the severityof the crash and thereby decide whether or not to deploy the airbag.They are thus using the occupant motion as a crash sensor. No mention ismade of determining the out-of-position status of the occupant or of anyof the other features of occupant monitoring as disclosed in one or moreof the above-referenced patents and patent applications. It isinteresting to note that nowhere does Mattes et al. discuss how to useactive or passive infrared to determine the position of the occupant. Aspointed out in one or more of the above-referenced patents and patentapplications, direct occupant position measurement based on passiveinfrared is probably not possible and, until very recently, was verydifficult and expensive with active infrared requiring the modulation ofan expensive GaAs infrared laser. Since there is no mention of theseproblems, the method of use contemplated by Mattes et al. must besimilar to the electric eye concept where position is measuredindirectly as the occupant passes by a plurality of longitudinallyspaced-apart sensors.

[0018] The object of an occupant out-of-position sensor is to determinethe location of the head and/or chest of the vehicle occupant relativeto the airbag since it is the impact of either the head or chest withthe deploying airbag which can result in serious injuries. Both White etal. and Mattes et al. describe only lower mounting locations of theirsensors in front of the occupant such as on the dashboard or below thesteering wheel. Both such mounting locations are particularly prone todetection errors due to positioning of the occupant's hands, arms andlegs. This would require at least three, and preferably more, suchsensors and detectors and an appropriate logic circuitry which ignoresreadings from some sensors if such readings are inconsistent withothers, for the case, for example, where the driver's arms are theclosest objects to two of the sensors.

[0019] White et al. also describe the use of error correction circuitry,without defining or illustrating the circuitry, to differentiate betweenthe velocity of one of the occupant's hands as in the case where he/sheis adjusting the knob on the radio and the remainder of the occupant.Three ultrasonic sensors of the type disclosed by White et al. might, insome cases, accomplish this differentiation if two of them indicatedthat the occupant was not moving while the third was indicating that heor she was. Such a combination, however, would not differentiate betweenan occupant with both hands and arms in the path of the ultrasonictransmitter at such a location that they were blocking a substantialview of the occupant's head or chest. Since the sizes and drivingpositions of occupants are extremely varied, it is now believed thatpattern recognition systems and preferably trained pattern recognitionsystems, such as neural networks, are required when a clear view of theoccupant, unimpeded by his/her extremities, cannot be guaranteed.

[0020] Fujita et al., in U.S. Pat. No. 5,074,583, describe anothermethod of determining the position of the occupant but do not use thisinformation to suppress deployment if the occupant is out-of-position.In fact, the closer the occupant gets to the airbag, the faster theinflation rate of the airbag is according to the Fujita et al. patent,which thereby increases the possibility of injuring the occupant. Fujitaet al. do not measure the occupant directly but instead determine his orher position indirectly from measurements of the seat position and thevertical size of the occupant relative to the seat (occupant height).This occupant height is determined using an ultrasonic displacementsensor mounted directly above the occupant's head.

[0021] As discussed above, the optical systems described herein are alsoapplicable for many other sensing applications both inside and outsideof the vehicle compartment such as for sensing crashes before they occuras described in U.S. Pat. No. 5,829,782, for a smart headlightadjustment system and for a blind spot monitor (also disclosed in U.S.provisional patent application Ser. No. 60/202,424).

[0022] 2. Definitions

[0023] Preferred embodiments of the invention are described below andunless specifically noted, it is the applicants' intention that thewords and phrases in the specification and claims be given the ordinaryand accustomed meaning to those of ordinary skill in the applicableart(s). If the applicant intends any other meaning, he will specificallystate he is applying a special meaning to a word or phrase.

[0024] Likewise, applicants' use of the word “function” here is notintended to indicate that the applicants seek to invoke the specialprovisions of 35 U.S.C. §112, sixth paragraph, to define theirinvention. To the contrary, if applicants wish to invoke the provisionsof 35 U.S.C. §112, sixth paragraph, to define their invention, they willspecifically set forth in the claims the phrases “means for” or “stepfor” and a function, without also reciting in that phrase any structure,material or act in support of the function. Moreover, even if applicantsinvoke the provisions of 35 U.S.C. §112, sixth paragraph, to definetheir invention, it is the applicants' intention that their inventionsnot be limited to the specific structure, material or acts that aredescribed in the preferred embodiments herein. Rather, if applicantsclaim their inventions by specifically invoking the provisions of 35U.S.C. §112, sixth paragraph, it is nonetheless their intention to coverand include any and all structure, materials or acts that perform theclaimed function, along with any and all known or later developedequivalent structures, materials or acts for performing the claimedfunction.

[0025] The use of pattern recognition is important to the instantinvention as well as to one or more of those disclosed in theabove-referenced patents and patent applications above. “Patternrecognition” as used herein will generally mean any system whichprocesses a signal that is generated by an object, or is modified byinteracting with an object, in order to determine which one of a set ofclasses that the object belongs to. Such a system might determine onlythat the object is or is not a member of one specified class, or itmight attempt to assign the object to one of a larger set of specifiedclasses, or find that it is not a member of any of the classes in theset. The signals processed are generally electrical signals coming fromtransducers which are sensitive to either acoustic or electromagneticradiation and, if electromagnetic, they can be either visible light,infrared, ultraviolet or radar or low frequency radiation as used incapacitive sensing systems.

[0026] A trainable or a trained pattern recognition system as usedherein means a pattern recognition system which is taught variouspatterns by subjecting the system to a variety of examples. The mostsuccessful such system is the neural network. Not all patternrecognition systems are trained systems and not all trained systems areneural networks. Other pattern recognition systems are based on fuzzylogic, sensor fusion, Kalman filters, correlation as well as linear andnon-linear regression. Still other pattern recognition systems arehybrids of more than one system such as neural-fuzzy systems.

[0027] A pattern recognition algorithm will thus generally mean analgorithm applying or obtained using any type of pattern recognitionsystem, e.g., a neural network, sensor fusion, fuzzy logic, etc.

[0028] To “identify” as used herein will usually mean to determine thatthe object belongs to a particular set or class. The class may be onecontaining, for example, all rear facing child seats, one containing allhuman occupants, or all human occupants not sitting in a rear facingchild seat depending on the purpose of the system. In the case where aparticular person is to be recognized, the set or class will containonly a single element, i.e., the person to be recognized.

[0029] To “ascertain the identity of” as used herein with reference toan object will generally mean to determine the type or nature of theobject (obtain information as to what the object is), i.e., that theobject is an adult, an occupied rear facing child seat, an occupiedfront facing child seat, an unoccupied rear facing child seat, anunoccupied front facing child seat, a child, a dog, a bag of groceries,a car, a truck, a tree, a pedestrian, a deer etc.

[0030] An “occupying item” or “occupant” of a seat or “object” in a seatmay be a living occupant such as a human being or a dog, another livingorganism such as a plant, or an inanimate object such as a box or bag ofgroceries.

[0031] A “rear seat” of a vehicle as used herein will generally mean anyseat behind the front seat on which a driver sits. Thus, in minivans orother large vehicles where there are more than two rows of seats, eachrow of seats behind the driver is considered a rear seat and thus theremay be more than one “rear seat” in such vehicles. The space behind thefront seat includes any number of such rear seats as well as any trunkspaces or other rear areas such as are present in station wagons.

[0032] An optical image will generally mean any type of image obtainedusing electromagnetic radiation including visual, infrared and radarradiation.

[0033] In the description herein on anticipatory sensing, the term“approaching” when used in connection with the mention of an object orvehicle approaching another will usually mean the relative motion of theobject toward the vehicle having the anticipatory sensor system. Thus,in a side impact with a tree, the tree will be considered as approachingthe side of the vehicle and impacting the vehicle. In other words, thecoordinate system used in general will be a coordinate system residingin the target vehicle. The “target” vehicle is the vehicle that is beingimpacted. This convention permits a general description to cover all ofthe cases such as where (i) a moving vehicle impacts into the side of astationary vehicle, (ii) where both vehicles are moving when theyimpact, or (iii) where a vehicle is moving sideways into a stationaryvehicle, tree or wall.

[0034] “Out-of-position” as used for an occupant will generally meanthat the occupant, either the driver or a passenger, is sufficientlyclose to an occupant protection apparatus (airbag) prior to deploymentthat he or she is likely to be more seriously injured by the deploymentevent itself than by the accident. It may also mean that the occupant isnot positioned appropriately in order to attain the beneficial,restraining effects of the deployment of the airbag. As for the occupantbeing too close to the airbag, this typically occurs when the occupant'shead or chest is closer than some distance such as about 5 inches fromthe deployment door of the airbag module. The actual distance whereairbag deployment should be suppressed depends on the design of theairbag module and is typically farther for the passenger airbag than forthe driver airbag.

[0035] 3. Pattern Recognition Prior Art

[0036] Japanese Patent No. 3-42337 (A) to Ueno discloses a device fordetecting the driving condition of a vehicle driver comprising a lightemitter for irradiating the face of the driver and a means for pickingup the image of the driver and storing it for later analysis. Means areprovided for locating the eyes of the driver and then the irises of theeyes and then determining if the driver is looking to the side orsleeping. Ueno determines the state of the eyes of the occupant ratherthan determining the location of the eyes relative to the other parts ofthe vehicle passenger compartment. Such a system can be defeated if thedriver is wearing glasses, particularly sunglasses, or another opticaldevice which obstructs a clear view of his/her eyes. Pattern recognitiontechnologies such as neural networks are not used.

[0037] U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rulesto isolate the eyes and mouth of a driver and uses this information topermit the driver to control the radio, for example, or other systemswithin the vehicle by moving his eyes and/or mouth. Ando uses naturallight and analyzes only the head of the driver. He also makes no use oftrainable pattern recognition systems such as neural networks, nor isthere any attempt to identify the contents of the vehicle nor of theirlocation relative to the vehicle passenger compartment. Rather, Ando islimited to control of vehicle devices by responding to motion of thedriver's mouth and eyes.

[0038] U.S. Pat. No. 5,298,732 to Chen also concentrates on locating theeyes of the driver so as to position a light filter between a lightsource such as the sun or the lights of an oncoming vehicle, and thedriver's eyes. Chen does not explain in detail how the eyes are locatedbut does supply a calibration system whereby the driver can adjust thefilter so that it is at the proper position relative to his or her eyes.Chen references the use of automatic equipment for determining thelocation of the eyes but does not describe how this equipment works. Inany event, there is no mention of illumination of the occupant,monitoring the position of the occupant, other that the eyes,determining the position of the eyes relative to the passengercompartment, or identifying any other object in the vehicle other thanthe driver's eyes. Also, there is no mention of the use of a trainablepattern recognition system.

[0039] U.S. Pat. No. 5,305,012 to Faris also describes a system forreducing the glare from the headlights of an oncoming vehicle. Farislocates the eyes of the occupant utilizing two spaced apart infraredcameras using passive infrared radiation from the eyes of the driver.Again, Faris is only interested in locating the driver's eyes relativeto the sun or oncoming headlights and does not identify or monitor theoccupant or locate the occupant relative to the passenger compartment orthe airbag. Also, Faris does not use trainable pattern recognitiontechniques such as neural networks. Faris, in fact, does not even sayhow the eyes of the occupant are located but refers the reader to a bookentitled Robot Vision (1991) by Berthold Horn, published by MIT Press,Cambridge, Mass. A review of this book did not appear to provide theanswer to this question. Also, Faris uses the passive infrared radiationrather than illuminating the occupant with active infrared radiation orin general electromagnetic radiation.

[0040] The use of neural networks as the pattern recognition technologyis important to several of the implementations of this invention sinceit makes the monitoring system robust, reliable and practical. Theresulting algorithm created by the neural network program is usuallyonly a few hundred lines of code written in the C or C++ computerlanguage as opposed to typically many hundreds of lines when thetechniques of the above patents to Ando, Chen and Faris are implemented.As a result, the resulting systems are easy to implement at a low cost,making them practical for automotive applications. The cost of the CCDand CMOS arrays, for example, have been prohibitively expensive untilrecently, rendering their use for VIMS impractical. Similarly, theimplementation of the techniques of the above referenced patentsrequires expensive microprocessors while the implementation with neuralnetworks and similar trainable pattern recognition technologies permitsthe use of low cost microprocessors typically costing less than $10 inlarge quantities.

[0041] The present invention preferably uses sophisticated trainablepattern recognition capabilities such as neural networks. Usually thedata is preprocessed, as discussed below, using various featureextraction techniques. An example of such a pattern recognition systemusing neural networks on sonar signals is discussed in two papers byGorman, R. P. and Sejnowski, T. J. “Analysis of Hidden Units in aLayered Network Trained to Classify Sonar Targets”, Neural Networks,Vol. 1. pp. 75-89, 1988, and “Learned Classification of Sonar TargetsUsing a Massively Parallel Network”, IEEE Transactions on Acoustics,Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples offeature extraction techniques can be found in U.S. Pat. No. 4,906,940entitled “Process and Apparatus for the Automatic Detection andExtraction of Features in Images and Displays” to Green et al. Examplesof other more advanced and efficient pattern recognition techniques canbe found in U.S. Pat. No. 5,390,136 entitled “Artificial Neuron andMethod of Using Same and U.S. Pat. No. 5,517,667 entitled “NeuralNetwork and Method of Using Same” to S. T. Wang. Other examples includeU.S. Pat. Nos. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744(Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S.Pat. No. 4,881,270 (Knecht et al). All of the above references areincorporated herein by reference. Neural networks as used herein includeall types of neural networks including modular neural networks, cellularneural networks and support vector machines and all combinations asdescribed in detail in U.S. Pat. No. 6,445,988, which is incorporatedherein by reference in its entirety, and referred to therein as“combination neural networks”.

[0042] 4. Optics

[0043] Optics can be used in several configurations for monitoring theinterior of a passenger compartment or exterior environment of anautomobile. In one known method, a laser optical system uses a GaAsinfrared laser beam to momentarily illuminate an object, occupant orchild seat, in the manner as described and illustrated in FIG. 8 of U.S.Pat. No. 5,829,782 referenced above. The receiver can be acharge-coupled device or CCD (a type of TV camera), or a CMOS imager toreceive the reflected light. The laser can either be used in a scanningmode, or, through the use of a lens, a cone of light can be createdwhich covers a large portion of the object. In these configurations, thelight can be accurately controlled to only illuminate particularpositions of interest within or around the vehicle. In the scanningmode, the receiver need only comprise a single or a few active elementswhile in the case of the cone of light, an array of active elements isneeded. The laser system has one additional significant advantage inthat the distance to the illuminated object can be determined asdisclosed in the commonly owned '462 patent as also described below.When a single receiving element is used, a PIN or avalanche diode ispreferred.

[0044] In a simpler case, light generated by a non-coherent lightemitting diode (LED) device is used to illuminate the desired area. Inthis case, the area covered is not as accurately controlled and a largerCCD or CMOS array is required. Recently, however, the cost of CCD andCMOS arrays has dropped substantially with the result that thisconfiguration may now be the most cost-effective system for monitoringthe passenger compartment as long as the distance from the transmitterto the objects is not needed. If this distance is required, then thelaser system, a stereographic system, a focusing system, a combinedultrasonic and optic system, or a multiple CCD or CMOS array system asdescribed herein is required. Alternately, a modulation system such asused with the laser distance system can be used with a CCD or CMOScamera and distance determined on a pixel by pixel basis.

[0045] A mechanical focusing system, such as used on some camera systemscan determine the initial position of an occupant but is too slow tomonitor his/her position during a crash. Although the example of anoccupant is used herein as an example, the same or similar principlesapply to objects exterior to the vehicle. A distance measuring systembased on focusing is described in U.S. Pat. No. 5,193,124 (Subbarao)which can either be used with a mechanical focusing system or with twocameras, the latter of which would be fast enough. Although the Subbaraopatent provides a good discussion of the camera focusing art and istherefore incorporated herein by reference, it is a more complicatedsystem than is needed for the practicing the instant invention. In fact,a neural network can also be trained to perform the distancedetermination based on the two images taken with different camerasettings or from two adjacent CCD's and lens having different propertiesas the cameras disclosed in Subbarao making this technique practical forthe purposes of this instant invention. Distance can also be determinedby the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by thespreading or defocusing of a pattern of structured light projected ontothe object of interest. Distance can also be measured by using time offlight measurements of the electromagnetic waves or by multiple CCD orCMOS arrays as is a principle teaching of this invention.

[0046] In each of these cases, regardless of the distance measurementsystem used; a trained pattern recognition system, as defined above, isused in the instant invention to identify and classify, and in somecases to locate, the illuminated object and its constituent parts.

[0047] 5. Optics and Acoustics

[0048] The laser systems described above are expensive due to therequirement that they be modulated at a high frequency if the distancefrom the airbag to the occupant, for example, needs to be measured.Alternately, modulation of another light source such as an LED can bedone and the distance measurement accomplished using a CCD or CMOS arrayon a pixel by pixel basis.

[0049] Both laser and non-laser optical systems in general are good atdetermining the location of objects within the two dimensional plane ofthe image and a pulsed laser radar system in the scanning mode candetermine the distance of each part of the image from the receiver bymeasuring the time of flight through range gating techniques. It is alsopossible to determine distance with the non-laser system by focusing asdiscussed above, or stereographically if two spaced apart receivers areused and, in some cases the mere location in the field of view can beused to estimate the position relative to the airbag, for example.Finally, a recently developed pulsed quantum well diode laser alsoprovides inexpensive distance measurements as discussed below.

[0050] Acoustic systems are additionally quite effective at distancemeasurements since the relatively low speed of sound permits simpleelectronic circuits to be designed and minimal microprocessor capabilityis required. If a coordinate system is used where the z axis is from thetransducer to the occupant, acoustics are good at measuring z dimensionswhile simple optical systems using a single CCD are good at measuring xand y dimensions. The combination of acoustics and optics,.therefore,permits all three measurements to be made from one location with lowcost components as discussed in commonly assigned U.S. Pat. Nos.5,845,000 and 5,835,613.

[0051] One example of a system using these ideas is an optical systemwhich floods the passenger seat with infrared light coupled with a lensand CCD or CMOS array which receives and displays the reflected lightand an analog to digital converter (ADC), or frame grabber, whichdigitizes the output of the CCD or CMOS and feeds it to a featureextraction algorithm (such as an edge detector) and then to anArtificial Neural Network (ANN) or other pattern recognition system foranalysis. This system uses an ultrasonic transmitter and receiver formeasuring the distances to the objects located in the passenger seat.The receiving transducer feeds its data into an ADC and from there theconverted data is directed into the ANN. The same ANN can be used forboth systems thereby providing full three-dimensional data for the ANNto analyze. This system, using low cost components, will permit accurateidentification and distance measurements. If a phased array system isadded to the acoustic part of the system, the optical part can determinethe location of the driver's ears, for example, and the phased array candirect a narrow beam to the location and determine the distance to theoccupant's ears.

[0052] Although the use of ultrasound for distance measurement has manyadvantages, it also has some drawbacks. First, the speed of sound limitsthe rate at which the position of the occupant can be updated toapproximately 10 milliseconds, which though sufficient for most cases,is marginal if the position of the occupant is to be tracked during avehicle crash. Second, ultrasound waves are diffracted by changes in airdensity that can occur when the heater or air conditioner is operated orwhen there is a high-speed flow of air past the transducer. Third, theresolution of ultrasound is limited by its wavelength and by thetransducers, which are high Q tuned devices. Typically, the resolutionof ultrasound is on the order of about 2 to 3 inches. Finally, thefields from ultrasonic transducers are difficult to control so thatreflections from unwanted objects or surfaces add noise to the data.

[0053] 6. Applications

[0054] The applications for this technology are numerous as described inthe patents and patent applications listed above. They include: (i) themonitoring of the occupant for safety purposes to prevent airbagdeployment induced injuries, (ii) the locating of the eyes of theoccupant (driver) to permit automatic adjustment of the rear viewmirror(s), (iii) the location of the seat to place the occupant's eyesat the proper position to eliminate the parallax in a heads-up displayin night vision systems, (iv) the location of the ears of the occupantfor optimum adjustment of the entertainment system, (v) theidentification of the occupant for security reasons, (vi) thedetermination of obstructions in the path of a closing door or window,(vii) the determination of the position of the occupant's shoulder sothat the seat belt anchorage point can be adjusted for the bestprotection of the occupant, (viii) the determination of the position ofthe rear of the occupants head so that the headrest can be adjusted tominimize whiplash injuries in rear impacts, (ix) anticipatory crashsensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii)sunlight and headlight glare reduction and many others. In fact, overforty products alone have been identified based on the ability toidentify and monitor objects and parts thereof in the passengercompartment of an automobile or truck. In addition, there are manyapplications of the apparatus and methods described herein formonitoring the environment exterior to the vehicle.

[0055] 7. Other Prior Art

[0056] European Patent Application No. 98110617.2 (Publication No. 0 885782 A1), corresponding to U.S. patent application Ser. No. 08/872,836filed Jun. 11, 1997, describes a purportedly novel motor vehicle controlsystem including a pair of cameras which operatively produce first andsecond images of a passenger area. A distance processor determines thedistances that a plurality of features in the first and second imagesare from the cameras based on the amount that each feature is shiftedbetween the first and second images. An analyzer processes thedetermined distances and determines the size of an object on the seat.Additional analysis of the distance also may determine movement of theobject and the rate of movement. The distance information also can beused to recognize predefined patterns in the images and this identifyobjects. An air bag controller utilizes the determined objectcharacteristics in controlling deployment of the air bag.

[0057] A paper entitled “Sensing Automobile Occupant Position withOptical Triangulation” by W. Chappelle, Sensors, December 1995,describes the use of optical triangulation techniques for determiningthe presence and position of people or rear-facing infant seats in thepassenger compartment of a vehicle in order to guarantee the safedeployment of an air bag. The paper describes a system called the“Takata Safety Shield” which purportedly makes high-speed distancemeasurements from the point of air bag deployment using a modulatedinfrared beam projected from an LED source. Two detectors are provided,each consisting of an imaging lens and a position-sensing detector.

[0058] A paper entitled “An Interior Compartment Protection System basedon Motion Detection Using CMOS Imagers” by S. B. Park et al., 1998 IEEEInternational Conference on Intelligent Vehicles, describes apurportedly novel image processing system based on a CMOS image sensorinstalled at the car roof for interior compartment monitoring includingtheft prevention and object recognition. One disclosed camera system isbased on a CMOS image sensor and a near infrared (NIR) light emittingdiode (LED) array.

[0059] A paper entitled “A 256×256 CMOS Brightness Adaptive ImagingArray with Column-Parallel Digital Output” by C. Sodini et al., 1988IEEE International Conference on Intelligent Vehicles, describes a CMOSimage sensor for intelligent transportation system applications such asadaptive cruise control and traffic monitoring. Among the purportednovelties is the use of a technique for increasing the dynamic range ina CMOS imager by a factor of approximately 20, which technique is basedon a previously described technique for CCD imagers.

[0060] A paper entitled “Intelligent System for Video Monitoring ofVehicle Cockpit” by S. Boverie et al., SAE Technical Paper Series No.980613, Feb. 23-26, 1998, describes the installation of anoptical/retina sensor in the vehicle and several uses of this sensor.Possible uses are said to include observation of the driver's face(eyelid movement) and the driver's attitude to allow analysis of thedriver's vigilance level and warn him/her about critical situations andobservation of the front passenger seat to allow the determination ofthe presence of somebody or something located on the seat and to valuethe volumetric occupancy of the passenger for the purpose of optimizingthe operating conditions for air bags.

[0061] Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an imageanalyzer (M5 in FIG. 1) for analyzing the position of driver includingan infrared light source which illuminates the driver's face and animage detector which receives light from the driver's face, determinesthe position of facial feature, e.g., the eyes in three dimensions, andthus determines the position of the driver in three dimensions. Apattern recognition process is used to determine the position of thefacial features and entails converting the pixels forming the image toeither black or white based on intensity and conducting an analysisbased on the white area in order to find the largest contiguous whitearea and the center point thereof Based on the location of the centerpoint of the largest contiguous white area, the driver's height isderived and a heads up display is adjusted so information is withindriver's field of view. The pattern recognition process can be appliedto detect the eyes, mouth, or nose of the driver based on thedifferentiation between the white and black areas.

[0062] Ando (U.S. Pat. No. 5,008,946) describes a system whichrecognizes an image and specifically ascertains the position of thepupils and mouth of the occupant to enable movement of the pupils andmouth to control electrical devices installed in the automobile. Thesystem includes a camera which takes a picture of the occupant andapplies algorithms based on pattern recognition techniques to analyzethe picture, converted into an electrical signal, to determine theposition of certain portions of the image, namely the pupils and mouth.

[0063] Masamori (U.S. Pat. No. 5,227,784) describes a system which isbased on radar, specifically it is a collision avoidance system aimed atdetecting vehicles which are at some distance from the vehicle.

[0064] Suzuki et al. (U.S. Pat. No. 5,026,153) describes a vehicletracking control for continuously detecting the distance and directionto a preceding vehicle irrespective of background dark/lightdistribution. In this system, every vehicle must have a light on itsrear that emits a constant or time varying signal and two photoelectricsensors that zero in on the light emitted from the preceding vehicle areused and thereby determine both the distance and angular position of thepreceding vehicle.

[0065] Krumm (U.S. Pat. No. 5,983,147) describes a system fordetermining the occupancy of a passenger compartment including a pair ofcameras mounted so as to obtain binocular stereo images of the samelocation in the passenger compartment. A representation of the outputfrom the cameras is compared to stored representations of knownoccupants and occupancy situations to determine which storedrepresentation the output from the cameras most closely approximates.The stored representations include that of the presence or absence of aperson or an infant seat in the front passenger seat.

[0066] Farmer et al. (U.S. Pat. No. 6,005,958) describes a method andsystem for detecting the type and position of a vehicle occupantutilizing a single camera unit. The single camera unit is positioned atthe driver or passenger side A-pillar in order to generate data of thefront seating area of the vehicle. The type and position of the occupantis used to optimize the efficiency and safety in controlling deploymentof an occupant protection device such as an air bag.

[0067] A paper by Rudolf Schwarte, et al. entitled “New Powerful SensoryTool in Automotive Safety Systems Based on PMD-Technology”, Eds. S.Krueger, W. Gessner, Proceedings of the AMAA 2000 Advanced Microsystemsfor Automotive Applications 2000, Springer Verlag; Berlin, Heidelberg,N.Y., ISBN 3-540-67087-4, describes an implementation of the teachingsof the instant invention wherein a modulated light source is used inconjunction with phase determination circuitry to locate the distance toobjects in the image on a pixel by pixel basis. This camera is an activepixel camera the use of which for internal and external vehiclemonitoring is also a teaching of this invention. The novel feature ofthe PMD camera is that the pixels are designed to provide a distancemeasuring capability within each pixel itself. This then is a novelapplication of the active pixel and distance measuring teachings of theinstant invention.

[0068] The instant invention as described in the above-referencedcommonly assigned patents and patent applications, teaches the use ofmodulating the light used to illuminate an object and to determine thedistance to that object based on the phase difference between thereflected radiation and the transmitted radiation. The illumination canbe modulated at a single frequency when short distances such as withinthe passenger compartment are to be measured. Typically, the modulationwavelength would be selected such that one wave would have a length ofapproximately one meter or less. This would provide resolution of 1 cmor less. For larger vehicles, a longer wavelength would be desirable.For measuring longer distances, the illumination can be modulated atmore than one frequency to eliminate cycle ambiguity if there is morethan one cycle between the source of illumination and the illuminatedobject. This technique is particularly desirable when monitoring objectsexterior to the vehicle to permit accurate measurements of devices thatare hundreds of meters from the vehicle as well as those that are a fewmeters away. Naturally, there are other modulation methods thateliminate the cycle ambiguity such as modulation with a code that isused with a correlation function to determine the phase shift or timedelay. This code can be a pseudo random number in order to permit theunambiguous monitoring of the vehicle exterior in the presence of othervehicles with the same system. This is sometimes known as noise radar,noise modulation (either of optical or radar signals), ultra wideband(UWB) or the techniques used in Micropower impulse radar (MIR).

[0069] Although a simple frequency modulation scheme has been disclosedso far, it is also possible to use other coding techniques including thecoding of the illumination with one of a variety of correlation patternsincluding a pseudo-random code. Similarly, although frequency and codedomain systems have been described, time domain systems are alsoapplicable wherein a pulse of light is emitted and the time of flightmeasured. Additionally, in the frequency domain case, a chirp can beemitted and the reflected light compared in frequency with the chirp todetermine by frequency difference the distance to the object. Althougheach of these techniques is known to those skilled in the art, they haveheretofore not been applied for monitoring objects within or outside ofa vehicle.

OBJECTS AND SUMMARY OF THE INVENTION

[0070] Principle objects and advantages of the optical sensing system inaccordance with the invention are:

[0071] 1. To provide a system for monitoring the environment exterior ofa vehicle in order to determine the presence and classification,identification and/or location of objects in the exterior environment.

[0072] 2. To provide a blind spot detector which detects and categorizesan object in the driver's blind spot or other location in the vicinityof the vehicle, and warns the driver in the event the driver begins tochange lanes, for example, or continuously informs the driver of thestate of occupancy of the blind spot.

[0073] 3. To provide a camera system for interior and exteriormonitoring, which can adjust on a pixel by pixel basis for the intensityof the received light.

[0074] 4. To provide for the use of an active pixel camera for interiorand exterior vehicle monitoring.

[0075] 5. To recognize the presence of a human on a particular seat of amotor vehicle and to use this information to affect the operation ofanother vehicle system such as the airbag, heating and air conditioning,or entertainment systems, among others.

[0076] 6. To recognize the presence of a human on a particular seat of amotor vehicle and then to determine his/her position and to use thisposition information to affect the operation of another vehicle system.

[0077] 7. To determine the position, velocity or size of an occupant ina motor vehicle and to utilize this information to control the rate ofgas generation, or the amount of gas generated by an airbag inflatorsystem.

[0078] 8. To determine the presence or position of rear seated occupantsin the vehicle and to use this information to affect the operation of arear seat protection airbag for frontal, side and/or rear impacts.

[0079] 9. To recognize the presence of a rear facing child seat on aparticular seat of a motor vehicle and to use this information to affectthe operation of another vehicle system such as the airbag system.

[0080] 10. To determine the approximate location of the eyes of a driverand to use that information to control the position of one or more ofthe rear view mirrors of the vehicle.

[0081] 11. To monitor the position of the head of the vehicle driver anddetermine whether the driver is falling asleep or otherwise impaired andlikely to lose control of the vehicle and to use that information toaffect another vehicle system.

[0082] 12. To provide an occupant position sensor which reliablypermits, and in a timely manner, a determination to be made that theoccupant is out-of-position, or will become out-of-position, and likelyto be injured by a deploying airbag and to then output a signal tosuppress the deployment of the airbag.

[0083] 13. To provide an anticipatory sensor that permits accurateidentification of the about-to-impact object in the presence of snowand/or fog whereby the sensor is located within the vehicle.

[0084] 14. To provide a smart headlight dimmer system which senses theheadlights from an oncoming vehicle or the tail lights of a vehicle infront of the subject vehicle and identifies these lights differentiatingthem from reflections from signs or the road surface and then sends asignal to dim the headlights.

[0085] 15. To provide an occupant position determination in asufficiently short time that the position of an occupant can be trackedduring a vehicle crash.

[0086] 16. To provide an occupant vehicle interior monitoring systemwhich is not affected by temperature or thermal gradients.

[0087] 17. To provide an occupant vehicle interior monitoring systemwhich has high resolution to improve system accuracy and permits thelocation of body parts of the occupant to be determined.

[0088] 18. To provide an occupant vehicle interior monitoring systemwhich reduces the glare from sunlight and headlights by imposing afilter between the eyes of an occupant and the light source.

[0089] 19. To provide a system for recognizing the identity of aparticular individual in the vehicle.

[0090] 20. To use the principles of time of flight to measure thedistance to an occupant or object exterior to the vehicle.

[0091] 21. To obtain a three dimensional image from a device at onelocation on a vehicle.

[0092] 22. To use pattern recognition techniques for analyzingthree-dimensional image data of occupants of a vehicle and objectsexterior to the vehicle.

[0093] 23. To provide a system of frequency domain modulation of theillumination of an object interior or exterior of a vehicle.

[0094] 24. To utilize code modulation such as with a pseudo random codeto permit the unambiguous monitoring of the vehicle exterior in thepresence of other vehicles with the same system.

[0095] 25. To use a chirp frequency modulation technique to aid indetermining the distance to an object interior or exterior of a vehicle.

[0096] 26. To utilize a correlation pattern modulation in a form of codedivision modulation for determining the distance of an object interioror exterior of a vehicle.

[0097] These and other objects and advantages will become apparent fromthe following description of the preferred embodiments of the vehicleidentification and monitoring system of this invention.

[0098] Briefly, in order to achieve at least one of the objects andpossibly others, a monitoring arrangement for monitoring an environmentexterior of a vehicle comprises at least one receiver arranged toreceive waves from the environment exterior of the vehicle which containinformation on any objects in the environment and generate a signalcharacteristic of the received waves, and a processor coupled to thereceiver(s) and comprising pattern recognition means for processing thesignal to provide a classification, identification and/or location ofthe exterior object. The pattern recognition means, such as a neuralcomputer or neural network and the like, apply a pattern recognitionalgorithm generated from data of possible exterior objects and patternsof received waves from the possible exterior objects. In this manner,one or more systems in the vehicle may be coupled to the processor suchthat the operation the system(s) is/are affected in response to theclassification, identification and/or location of the exterior object.

[0099] This affected system may be a display viewable by the driver andarranged to show an image or icon of the exterior object or objects. Theaffected system may also be an internally deployed or externallydeployed airbag, deployment of which is controlled based on theidentification of the exterior object.

[0100] A pair of receivers spaced apart from one another may be used andthe receiver may be arranged to receive infrared waves. If a transmitteris provided to transmit waves into the environment exterior of thevehicle, then the receiver is arranged to receive waves transmitted bythe transmitter and reflected by any exterior objects.

[0101] A measurement system may be provided to measure a distancebetween the exterior object and the vehicle. The measurement system maybe a radar or laser radar system.

[0102] Another monitoring arrangement for monitoring an environmentexterior of a vehicle in accordance with the invention comprises atleast one CCD array positioned to obtain images of the environmentexterior of the vehicle and a processor coupled to the CCD array(s) andcomprising pattern recognition means for processing the images obtainedby the CCD array(s) to provide a classification, identification and/orlocation of the exterior object. In this manner, one or more systems inthe vehicle may be coupled to the processor such that the operation thesystem(s) is/are affected in response to the classification,identification and/or location of the exterior object. Optionally, thepattern recognition means apply a pattern recognition algorithmgenerated from data of possible exterior objects and images from the CCDarray(s) of the possible exterior objects. The same enhancementsdescribed for the arrangement above can be used in this arrangement aswell.

[0103] In another embodiment of the invention, at least one receiver isarranged on a rear view mirror of the vehicle to receive waves from theenvironment exterior of the vehicle which contain information on anyobjects in the environment and generate a signal characteristic of thereceived waves and a processor is coupled to the receiver(s) andarranged to classify, identify and/or locate the exterior object basedon the signal. As such, the operation of one or more systems in thevehicle, coupled to the processor, can be affected in response to theclassification, identification and/or location of the exterior object.The same enhancements described for the arrangements above can be usedin this embodiment as well.

[0104] Instead of being arranged on the rear view mirror, the receivercan be arranged on a B-pillar of the vehicle to receive waves from theenvironment exterior of and on the side of the vehicle which containinformation on any objects in the environment and generate a signalcharacteristic of the received waves. In addition, the receiver could bearranged in a door window trim panel of the vehicle to receive wavesfrom the environment exterior of the vehicle which contain informationon any objects in the environment and generate a signal characteristicof the received waves. Other mounting locations include in a rear windowand in a C-pillar of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The following drawings are illustrative of embodiments of theinvention and are not meant to limit the scope of the invention asencompassed by the claims.

[0106]FIG. 1A is a side planar view, with certain portions removed orcut away, of a portion of the passenger compartment of a vehicle showingseveral preferred mounting locations of interior vehicle monitoringsensors shown particularly for sensing the vehicle driver illustratingthe wave pattern from a CCD or CMOS optical position sensor mountedalong the side of the driver or centered above his or her head.

[0107]FIG. 1B is a view as in FIG. 1A illustrating the wave pattern froman optical system using an infrared light source and a CCD or CMOS arrayreceiver using the windshield as a reflection surface and showingschematically the interface between the vehicle interior monitoringsystem of this invention and an instrument panel mounted inattentivenesswarning light or buzzer and reset button.

[0108]FIG. 1C is a view as in FIG. 1A illustrating the wave pattern froman optical system using an infrared light source and a CCD or CMOS arrayreceiver where the CCD or CMOS array receiver is covered by a lenspermitting a wide angle view of the contents of the passengercompartment.

[0109]FIG. 1D is a view as in FIG. 1A illustrating the wave pattern froma pair of small CCD or CMOS array receivers and one infrared transmitterwhere the spacing of the CCD or CMOS arrays permits an accuratemeasurement of the distance to features on the occupant.

[0110]FIG. 1E is a view as in FIG. 1A illustrating the wave pattern froma set of ultrasonic transmitter/receivers where the spacing of thetransducers and the phase of the signal permits an accurate focusing ofthe ultrasonic beam and thus the accurate measurement of a particularpoint on the surface of the driver.

[0111]FIG. 2A is a side view, with certain portions removed or cut away,of a portion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors

[0112]FIG. 2B is a perspective view, with certain portions removed orcut away, of a portion of the passenger compartment of a vehicle showingsome preferred mounting locations of optical interior vehicle monitoringsensors.

[0113]FIG. 3 is a circuit schematic illustrating the use of the vehicleinterior monitoring sensor used as an occupant position sensor inconjunction with the remainder of the inflatable restraint system.

[0114]FIG. 4 is a schematic illustrating the circuit of an occupantposition-sensing device using a modulated infrared signal, beatfrequency and phase detector system.

[0115]FIG. 5 is a side planer view with parts cutaway and removed of avehicle showing the passenger compartment containing a driver and apreferred mounting location for an optical occupant position sensor foruse in side impacts and also of an optical rear of occupant's headlocator for use with a headrest adjustment system to reduce whiplashinjuries in rear impact crashes.

[0116]FIG. 6 is a side plan view of the interior of an automobile, withportions cut away and removed, with two optical occupant heightmeasuring sensors, one mounted into the headliner above the occupant'shead and the other mounted onto the A-pillar and also showing a seatbeltassociated with the seat where the seatbelt has an adjustable upperanchorage point which is automatically adjusted corresponding to theheight of the occupant.

[0117]FIG. 7 is a perspective view of a vehicle about to impact the sideof another vehicle showing the location of the various parts of theanticipatory sensor system of this invention.

[0118]FIG. 7A is an enlarged view of the section designated 7A in FIG.7.

[0119]FIG. 8 is a side planar view, with certain portions removed or cutaway, of a portion of the passenger compartment illustrating a sensorfor sensing the headlights of an oncoming vehicle and/or the taillightsof a leading vehicle used in conjunction with an automatic headlightdimming system.

[0120]FIG. 9 is a side planar view with parts cutaway and removed of asubject vehicle and an oncoming vehicle, showing the headlights of theoncoming vehicle and the passenger compartment of the subject vehicle,containing detectors of the driver's eyes and detectors for theheadlights of the oncoming vehicle and the selective filtering of thelight of the approaching vehicle's headlights through the use of aliquid crystal filter in the windshield.

[0121]FIG. 9A is an enlarged view of the section designated 9A in FIG.9.

[0122]FIG. 10 is a schematic illustration of a system for controllingoperation of a vehicle or a component thereof based on recognition of anauthorized individual.

[0123]FIG. 11 is a schematic illustration of a method for controllingoperation of a vehicle based on recognition of an individual.

[0124]FIG. 12 is a schematic illustration of the environment monitoringin accordance with the invention.

[0125]FIG. 13 is a flow chart of the environment monitoring inaccordance with the invention.

[0126]FIG. 14 is a schematic illustration of the position measuring inaccordance with the invention.

[0127]FIG. 15 is a schematic illustration of the exterior monitoringsystem in accordance with the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0128] Referring now to the drawings wherein the same reference numeralsrefer to the same or similar elements, a section of the passengercompartment of an automobile is shown generally as 100 in FIGS. 1A-1D. Adriver 101 of a vehicle sits on a seat 102 behind a steering wheel 103,which contains an airbag assembly 104. Airbag assembly 104 may beintegrated into the steering wheel assembly or coupled to the steeringwheel 103. Five transmitter and/or receiver assemblies 110, 111, 112,113 and 114 are positioned at various places in the passengercompartment (the specific locations of which are set forth below) todetermine the location of various parts of the driver, e.g., the head,chest and torso, relative to the airbag and to otherwise monitor theinterior of the passenger compartment. Monitoring of the interior of thepassenger compartment can entail detecting the presence or absence ofthe driver and passengers, differentiating between animate and inanimateobjects, detecting the presence of occupied or unoccupied child seats,rear-facing or forward-facing, and identifying and ascertaining theidentity of the occupying items in the passenger compartment. Processormeans such as control circuitry 120 is connected to thetransmitter/receiver assemblies 110-114 and controls the transmissionfrom the transmitters, if a transmission component is present in theassemblies, and captures the return signals from the receivers, if areceiver component is present in the assemblies. Control circuitry 120usually contains analog to digital converters (ADCs) or a frame grabber,a microprocessor containing sufficient memory and appropriate softwareincluding pattern recognition algorithms, and other appropriate drivers,signal conditioners, signal generators, etc. Usually, in any givenimplementation, only three or four of the transmitter/receiverassemblies would be used depending on their mounting locations asdescribed below.

[0129] With respect to the connection between the transmitter/receiverassemblies 110-114 and the control circuitry 120, a portion of thisconnection is shown as wires. It should be understood that all of theconnections between the transmitter/receiver assemblies 110-114 and thecontrol circuitry 120 may be wires, either individual wires leading fromthe control circuitry 120 to each of the transmitter/receiver assemblies110-114 or one or more wire buses.

[0130] With respect to the position of the control circuitry 120 in thedashboard of the vehicle, this position is for illustration purposesonly and does not limit the location of the control circuitry 120.Rather, the control circuitry 120 may be located anywhere convenient ordesired in the vehicle.

[0131] It is contemplated that a system and method in accordance withthe invention can include a single transmitter and multiple receivers,each at a different location. Thus, each receiver would not beassociated with a transmitter forming transmitter/receiver assemblies.Rather, for example, with reference to FIG. 1A, only element 110 wouldconstitute a transmitter/receiver assembly and elements 111, 112, 113,114 would be receivers only.

[0132] On the other hand, it is conceivable that in someimplementations, a system and method in accordance with the inventioninclude a single receiver and multiple transmitters. Thus, eachtransmitter would not be associated with a receiver formingtransmitter/receiver assemblies. Rather, for example, with reference toFIG. 1A, only element 110 would constitute a transmitter/receiverassembly and elements 111, 112, 113, 114 would be transmitters only.

[0133]FIG. 1A illustrates a typical wave pattern of transmitted infraredwaves from transmitter/receiver assembly 111, which is mounted on theside of the vehicle passenger compartment above the front, driver's sidedoor. Transmitter/receiver assembly 114, shown overlaid ontotransmitter/receiver 111, is actually mounted in the center headliner ofthe passenger compartment (and thus between the driver's seat and thefront passenger seat), near the dome light, and is aimed toward thedriver. Typically there will be symmetrical installation for thepassenger side of the vehicle. That is, a transmitter/receiver assemblywould be arranged above the front, passenger side door and anothertransmitter/receiver assembly would be arranged in the center headliner,near the dome light, and aimed toward the front, passenger side door.

[0134] In a preferred embodiment, each transmitter/receiver assembly111,114 comprises an optical transducer that will generally be used inconjunction with another optical transmitter/receiver assembly such asshown at 110, 112 and 113, which act in a similar manner. These opticaltransmitter/receiver assemblies are comprised of an optical transmitter,which may be an infrared LED (or possibly a near infrared (NIR) LED), alaser with a diverging lens or a scanning laser assembly, and a receiversuch as a CCD or CMOS array and particularly an active pixel CMOS cameraor array or a HDRL or HDRC camera or array as discussed below. Thetransducer assemblies map the location of the occupant(s), objects andfeatures thereof, in a two or three-dimensional image as will now bedescribed in more detail.

[0135] An active pixel camera is a special camera which has the abilityto adjust the sensitivity of each pixel of the camera similar to themanner in which an iris adjusts the sensitivity of a camera. Thus, theactive pixel camera automatically adjusts to the incident light on apixel-by-pixel basis. An active pixel camera differs from an activeinfrared sensor in that an active infrared sensor, such as of the typeenvisioned by Mattes et al. (discussed above), is generally a singlepixel sensor that measures the reflection of infrared light from anobject. In some cases, as in the HDRC camera, the output of each pixelis a logarithm of the incident light thus giving a high dynamic range tothe camera. This is similar to the technique used to suppress theeffects of thermal gradient distortion of ultrasonic signals asdescribed in the above cross-referenced patents. Thus if the incidentradiation changes in magnitude by 1,000,000, for example, the output ofthe pixel may change by a factor of only 6.

[0136] A dynamic pixel camera is a camera having a plurality of pixelsand which provides the ability to pick and choose which pixels should beobserved, as long as they are contiguous.

[0137] An HDRC camera is a type of active pixel camera where the dynamicrange of each pixel is considerably broader. An active pixel cameramanufactured by the Photobit Corporation has a dynamic range of 70 dbwhile an IMS Chips camera, an HDRC camera manufactured by anothermanufacturer, has a dynamic range of 120 db. Thus, the HDRC camera has a100,000 times greater range of light sensitivity than the Photobitcamera.

[0138] In a preferred implementation, four transducer assemblies arepositioned around the seat to be monitored, each comprising an LED witha diverging lens and a CMOS array. Although illustrated together, theilluminating source in many cases will not be co-located with thereceiving array. The LED emits a controlled angle, 120° for example,diverging cone of infrared radiation that illuminates the occupant fromboth sides and from the front and rear. This angle is not to be confusedwith the field angle used in ultrasonic systems. With ultrasound,extreme care is required to control the field of the ultrasonic waves sothat they will not create multipath effects and add noise to the system.With infrared, there is no reason, in the implementation now beingdescribed, other than to make the most efficient use of the infraredenergy, why the entire vehicle cannot be flooded with infrared energyeither from many small sources or from a few bright ones.

[0139] The image from each array is used to capture two dimensions ofoccupant position information, thus, the array of assembly 110positioned on the A-pillar, which is approximately 25% of the waylaterally across the headliner in front of the driver, provides a bothvertical and transverse information on the location of the driver. Asimilar view from the rear is obtained from the array of assembly 113positioned behind the driver on the roof of the vehicle and above theseatback potion of the seat 102. As such, assembly 113 also providesboth vertical and transverse information on the location of the driver.Finally, arrays of assemblies 111 and 114 provide both vertical andlongitudinal driver location information. Another preferred location isthe headliner centered directly above the seat of interest. The positionof the assemblies 110-114 may differ from that shown in the drawings. Inthe invention, in order that the information from two or more of theassemblies 110-114 may provide a three-dimensional image of theoccupant, or portion of the passenger compartment, the assemblies shouldnot be arranged side-by-side. A side-by-side arrangement as used inseveral prior art references discussed above, will provide twoessentially identical views with the difference being a lateral shift.This does not enable a complete three-dimensional view of the occupant.

[0140] If each receiving array of assemblies 110,111,113,114 contains amatrix of 100 by 100 pixels, then 40,000 (4×100×100) pixels or dataelements of information will be created each time the systeminterrogates the driver seat, for example. There are many pixels of eachimage that can be eliminated as containing no useful information. Thistypically includes the comer pixels, back of the seat and other areaswhere an occupant cannot reside. This pixel pruning can typically reducethe number of pixels by up to 50 percent resulting in approximately20,000 remaining pixels. The output from each array is then comparedwith a series of stored arrays representing different unoccupiedpositions of the seat, seatback, steering wheel etc. For each array,each of the stored arrays is subtracted from the acquired array and theresults analyzed to determine which subtraction resulted in the bestmatch. The best match is determined by such things as the total numberof pixels reduced below the threshold level, or the minimum number ofremaining detached pixels, etc. Once this operation is completed for allfour images, the position of the movable elements within the passengercompartment has been determined. This includes the steering wheel angle,telescoping position, seatback angle, headrest position, and seatposition. This information can be used elsewhere by other vehiclesystems to eliminate sensors that are currently being used to sense suchpositions of these complements. Alternately, the sensors that arecurrently on the vehicle for sensing these complement positions can beused to simplify processes described above.

[0141] Each receiving array may also be a 256×256 CMOS pixel array asdescribed in the paper by C. Sodini et al. referenced above.

[0142] An alternate technique of differentiating between the occupantand the vehicle is to use motion. If the images of the passenger seatare compared over time, reflections from fixed objects will remainstatic whereas reflections from vehicle occupants will move. Thismovement can be used to differentiate the occupant from the background.

[0143] Following the subtraction process described above, each image nowconsists of typically as many as 50 percent fewer pixels leaving a totalof approximately 10,000 pixels remaining. The resolution of the imagesin each array can now be reduced by combining adjacent pixels andaveraging the pixel values. This results in a reduction to a total pixelcount of approximately 1000. The matrices of information that containsthe pixel values is now normalized to place the information in alocation in the matrix which is independent of the seat position. Theresulting normalized matrix of 1000 pixel values is now used as inputinto an artificial neural network and represents the occupancy of theseat independent of the position of the occupant.

[0144] The neural network has been previously trained on a significantnumber of occupants of the passenger compartment. The number of suchoccupants depends strongly on whether the driver or the passenger seatis being analyzed. The variety of seating states or occupancies of thepassenger seat is vastly greater than that of the driver seat. For thedriver seat, a typical training set will consist of approximately 100different vehicle occupancies. For the passenger seat, this number canexceed 1000. These numbers are used for illustration purposes only andwill differ significantly from vehicle model to vehicle model.

[0145] The neural network is now used to determine which of the storedoccupancies most closely corresponds to the measured data. The output ofthe neural network is an index of the setup that was used duringtraining that most closely matches the current measured state. Thisindex is used to locate stored information from the matched trainedoccupancy. Information that has been stored for the trained occupancytypically includes the locus of the centers of the chest and head of thedriver, as well as the approximate radius of pixels which is associatedwith this center to define the head area, for example. For the case ofFIG. 1A, it is now known from this exercise where the head, chest, andperhaps the eyes and ears, of the driver are most likely to be locatedand also which pixels should be tracked in order to know the preciseposition of the driver's head and chest. What has been described aboveis the identification process.

[0146] The normalization process conducted above created a displacementvalue for each of the CCD or CMOS arrays in the four assemblies110,111,113,114 which can now be used in reverse to find the preciselocation of the driver's head and chest or chest, for example, relativeto the known location of the airbag. From the vehicle geometry, and thehead and chest location information, a choice can now be made as towhether to track the head or chest for dynamic out-of-position.

[0147] Tracking of the motion of the occupant's head or chest can bedone using a variety of techniques. One preferred technique is to usedifferential motion, that is, by subtracting the current image from theprevious image to determine which pixels have changed in value and bylooking at the leading edge of the changed pixels and the width of thechanged pixel field, a measurement of the movement of the pixels ofinterest, and thus the driver, can be readily accomplished. Alternately,a correlation function can be derived which correlates the pixels in theknown initial position of the head, for example, with pixels that werederived from the latest image. The displacement of the center of thecorrelation pixels would represent the motion of the head of theoccupant. Naturally, a wide variety of other techniques will be nowobvious to those skilled in the art.

[0148] There are many mathematical techniques that can be applied tosimplify the above process. One technique used in military patternrecognition, for example, uses the Fourier transform of particular areasin an image to match with known Fourier transforms of known images. Inthis manner, the identification and location can be determinedsimultaneously. There is even a technique used for target identificationwhereby the Fourier transforms are compared optically. Other techniquesutilize thresholding to limit the pixels that will be analyzed by any ofthese processes. Other techniques search for particular features andextract those features and concentrate merely on the location of certainof these features. (See for example the Kage et al. artificial retinapublication referenced above which, together with the references citedtherein, is incorporated herein by reference.)

[0149] Generally, the pixel values are not directly fed into a patternrecognition system but rather the image is preprocessed through avariety of feature extraction techniques such as an edge detectionalgorithm. Once the edges are determined, a vector is created containingthe location of the edges and their orientation and that vector is fedinto the neural network, for example, which performs the patternrecognition.

[0150] The principal used in this preferred implementation of theinvention is to use images of different views of the occupant tocorrelate with known images that were used to train a neural network forvehicle occupancy. Then carefully measured positions of the known imagesare used to locate particular parts of the occupant such as his or herhead, chest, eyes, ears, mouth, etc. An alternate approach is to make athree-dimensional map of the occupant and to precisely locate thesefeatures using neural networks, sensor fusion, fuzzy logic or otherrules. One method of obtaining a three-dimensional map is to utilize ascanning laser radar system where the laser is operated in a pulse modeand the distance from the object being illuminated is determined usingrange gating in a manner similar to that described in various patents onmicropower impulse radar to McEwan. (See, for example, U.S. Pat. Nos.5,457,394 and 5,521,600).

[0151] The scanning portion of the pulse laser radar device can beaccomplished using rotating mirrors, mechanical motors, or preferably, asolid state system, for example one utilizing TeO₂ as an opticaldiffraction crystal with lithium niobate crystals driven by ultrasound(although other solid state systems not necessarily using TeO₂ andlithium niobate crystals could also be used). An alternate method is touse a micromachined mirror, which is supported at its center and causedto deflect by miniature coils. Such a device has been used to providetwo-dimensional scanning to a laser. This has the advantage over theTeO₂-lithium niobate technology in that it is inherently smaller andlower cost and provides two-dimensional scanning capability in one smalldevice. The maximum angular deflection that can be achieved with thisprocess is on the order of about 10 degrees. Thus, a diverging lens willbe needed for the scanning system.

[0152] An alternate method of obtaining three-dimensional informationfrom a scanning laser system is to use multiple arrays to replace thesingle arrays used in FIG. 1A. In the case, the arrays are displacedfrom each other and, through triangulation, the location of thereflection from the illumination by a laser beam of a point on theobject can be determined in a manner that is understood by those skilledin the art.

[0153] One important point concerns the location and number of opticalassemblies. It is possible to use fewer than four such assemblies with aresulting loss in accuracy. The number of four was chosen so that eithera forward or rear assembly or either of the side assemblies can beblocked by a newspaper, for example, without seriously degrading theperformance of the system. Since drivers rarely are reading newspaperswhile driving, fewer than four arrays are usually adequate for thedriver side.

[0154] The particular locations of the optical assemblies were chosen togive the most accurate information as to the locations of the occupant.This is based on an understanding of what information can be bestobtained from a visual image. There is a natural tendency on the part ofhumans to try to gauge distance from the optical sensors directly. This,as can be seen above, is at best complicated involving focusing systems,stereographic systems, multiple arrays and triangulation, time of flightmeasurement, etc. What is not intuitive to humans is to not try toobtain this distance directly from apparatus or techniques associatedwith the mounting location. Whereas ultrasound is quite good formeasuring distances from the transducer (the z-axis), optical systemsare better at measuring distances in the vertical and lateral directions(the x and y-axes). Since the precise locations of the opticaltransducers are known, that is, the geometry of the transducer locationsis known relative to the vehicle, there is no need to try to determinethe displacement of an object of interest from the transducer (thez-axis) directly. This can more easily done indirectly by anothertransducer. That is, the z-axis to one transducer is the x-axis toanother.

[0155] Ultrasonic transducers are relatively good at measuring thedistance along a radius to a reflective object. An optical array, suchas disclosed herein, on the other hand, can get accurate measurements intwo dimensions, the lateral and vertical dimensions relative to thetransducer. If we assume that the optical array has dimensions of 100 by100 as compared to an ultrasonic sensor that has a single dimension of100, an optical array can give therefore 100 times as much informationas the ultrasonic array. Most importantly, this vastly greater amount ofinformation does not cost significantly more to obtain than theinformation from the ultrasonic sensor.

[0156] As illustrated in FIGS. 1A-1D, the optical sensors are typicallylocated at the positions where the desired information is available withthe greatest resolution. These positions are typically in the centerfront and center rear of the occupancy seat and at the center on eachside and top. This is in contrast to the optimum location for ultrasonicsensors, which are the comers of such a rectangle that outlines theseated volume.

[0157] Systems based on ultrasonics and neural networks have been verysuccessful in analyzing the seated state of both the passenger anddriver seats of automobiles. Such systems are now going into productionfor preventing airbag deployment when a rear facing child seat or andout-of-position occupant is present. The ultrasonic systems, however,suffer from certain natural limitations that prevent the system accuracyfrom getting better than about 99 percent. These limitations relate tothe fact that the wavelength of ultrasound is typically between 3 and 8mm. As a result, unexpected results occur which are due partially to theinterference of reflections from different surfaces. Additionally,commercially available ultrasonic transducers are tuned devices thatrequire several cycles before they transmit significant energy andsimilarly require several cycles before they effectively receive thereflected signals. This requirement has the effect of smearing theresolution of the ultrasound to the point that, for example, using aconventional 40 kHz transducer, the resolution of the system isapproximately three inches.

[0158] In contrast, the wavelength of infrared is less than one micronand no significant interferences occur. Similarly, the system is nottuned and therefore is theoretically sensitive to a very few cycles. Asa result, resolution of the optical system is determined by the pixelspacing in the CCD or CMOS arrays. For this application, typical arrayshave been chosen to be 100 pixels by 100 pixels and therefore the spacebeing imaged can be broken up into pieces that are significantly lessthan 1 cm in size. Naturally, if greater resolution is required arrayshaving larger numbers of pixels are readily available. Another advantageof optical systems is that special lenses can be used to magnify thoseareas where the information is most critical and operate at reducedresolution where this is not the case. For example, the area closest tothe at-risk zone in front of the airbag can be magnified. This is notpossible with ultrasonic systems.

[0159] To summarize, although ultrasonic neural network systems areoperating with high accuracy, they do not totally eliminate the problemof deaths and injuries caused by airbag deployments. Optical systems, onthe other hand, at little increase in cost, have the capability ofvirtually 100 percent accuracy. Additional problems of ultrasonicsystems arise from the slow speed of sound and diffraction caused byvariations is air density. The slow sound speed limits the rate at whichdata can be collected and thus eliminates the possibility of trackingthe motion of an occupant during a high speed crash.

[0160] In the case of FIG. 1A, transmitter/receiver assemblies 110-114emit infrared waves that reflect off of the head and chest of the driverand return thereto. Periodically, the device, as commanded by controlcircuitry 120, transmits a pulse of infrared waves and the reflectedsignal is detected by the same or a different device. The transmitterscan either transmit simultaneously or sequentially. An associatedelectronic circuit and algorithm in control circuitry 120 processes thereturned signals as discussed above and determines the location of theoccupant in the passenger compartment. This information is then sent tothe crash sensor and diagnostic circuitry, which may also be resident incontrol circuitry 120 (programmed within a control module), whichdetermines if the occupant is close enough to the airbag that adeployment might, by itself, cause injury which exceeds that which mightbe caused by the accident itself In such a case, the circuit disablesthe airbag system and thereby prevents its deployment. In an alternatecase, the sensor algorithm assesses the probability that a crashrequiring an airbag is in process and waits until that probabilityexceeds an amount that is dependent on the position of the occupant.Thus, for example, the sensor might decide to deploy the airbag based ona need probability assessment of 50%, if the decision must be madeimmediately for an occupant approaching the airbag, but might wait untilthe probability rises to 95% for a more distant occupant. In thealternative, the crash sensor and diagnostic circuitry optionallyresident in control circuitry 120 may tailor the parameters of thedeployment (time to initiation of deployment, rate of inflation, rate ofdeflation, deployment time, etc.) based on the current position andpossibly velocity of the occupant, e.g., a depowered deployment.

[0161] Although a driver system has been illustrated, the front and rearseat passenger systems would be similar.

[0162] In another implementation, the sensor algorithm may determine therate that gas is generated to affect the rate that the airbag isinflated. In all of these cases, the position of the occupant is used toaffect the deployment of the airbag as to whether or not it should bedeployed at all, the time of deployment and/or the rate of inflation.

[0163] It should be understood that although the above descriptionmentions that the airbag system can be controlled by the controlcircuitry 120, any vehicular system, component or subsystem can becontrolled based on the information or data obtained bytransmitter/receiver assemblies 110-114. Control circuitry 120 can beprogrammed or trained, if for example a neural network is used, tocontrol heating an air-conditioning systems based on the presence ofoccupants in certain positions so as to optimize the climate control inthe vehicle. The entertainment system can also be controlled to providesound only to locations at which occupants are situated. There is nolimit to the number and type of vehicular systems, components andsubsystems that can be controlled using the image analysis techniquesdescribed herein.

[0164] Furthermore, if multiple vehicular systems are to be controlledby control circuitry 120, then these systems can be controlled by thecontrol circuitry 120 based on the status of particular components ofthe vehicle. For example, an indication of whether a key is in theignition can be used to direct the control circuitry 120 to eithercontrol an airbag system (when the key is present in the ignition) or anantitheft system (when the key is not present in the ignition). Controlcircuitry 120 would thus be responsive to the status of the ignition ofthe motor vehicle to perform one of a plurality of different functions.More particularly, the pattern recognition algorithm, such as the neuralnetwork described herein, could itself be designed to perform in adifferent way depending on the status of a vehicular component such asthe detected present of a key in the ignition. It could provide oneoutput to control an antitheft system when a key is not present andanother output when a key is present using the same inputs from thetransmitter/receiver assemblies 110-114.

[0165] The algorithm in control circuitry 120 can also be designed todetermine the location of the occupant's eyes either directly orindirectly through a determination of the location of the occupant andan estimation of the position of the eyes therefrom. As such, theposition of the rear view mirror 105 can be adjusted to optimize thedriver's use thereof.

[0166] Weight sensors 130 are also included in the system shown in FIG.1A. Although strain gage type sensors are schematically illustratedmounted to the supporting structure of the seat 102, any other type ofweight sensor can be used. Strain gage weight sensors are described indetail in U.S. patent application Ser. No. 09/193,209 that isincorporated herein by reference as if it were entirely set forthherein. Weight can be used to confirm the occupancy of the seat, i.e.,the presence or absence of an occupant as well as whether the seat isoccupied by a light or heavy object. In the latter case, a measuredweight of less than 60 pounds is often determinative of the presence ofa child seat whereas a measured weight of greater than 60 pounds isoften indicative of the absence of a child seat. The weight sensors 130can also be used to determine the weight distribution of the occupant ofthe seat and thereby ascertain whether the occupant is moving and theposition of the occupant. As such, the weight sensors 130 could be usedto confirm the position of the occupant. The measured weight ordistribution thereof can also be used in combination with the data fromthe transmitter/receiver assemblies 110-114 to provide an identificationof the occupants in the seat.

[0167] The accuracy of the optical occupant sensor is criticallydependent upon the accuracy of the camera. The dynamic range of lightwithin a vehicle exceeds 120 decibels. When a car is driving at night,for example, very little light is available whereas when driving in abright sunlight, especially in a convertible, the light intensity canoverwhelm most cameras. Additionally, the camera must be able to adjustrapidly to changes and light caused by, for example, the emergence ofthe vehicle from tunnel, or passing by other obstructions such as trees,buildings, other vehicles, etc. which temporarily block the sun andcause a strobing effect at frequencies approaching 1 kHz.

[0168] Recently, improvements have been made to CMOS cameras that havesignificantly increased their dynamic range. New logarithmic highdynamic range technology such as developed by IMS Chips of Stuttgart,Germany, is now available in HDRC (High Dynamic Range CMOS) cameras.This technology provides a 120 dB dynamic intensity response at eachpixel in a mono chromatic mode. The technology has a 1 million to onedynamic range at each pixel. This prevents blooming, saturation andflaring normally associated with CMOS and CCD camera technology. Thissolves a problem that will be encountered in an automobile when goingfrom a dark tunnel into bright sunlight. Such a range would even exceedthe 120 dB intensity.

[0169] There is also significant infrared radiation from bright sunlightand from incandescent lights within the vehicle. Such situations mayeven exceed the dynamic range of the HDRC camera and additionalfiltering may be required. Changing the bias on the receiver array, theuse of a mechanical iris, or of electrochromic glass or liquid crystalcan provide this filtering on a global basis but not at a pixel level.Filtering can also be used with CCD arrays, but the amount of filteringrequired is substantially greater than for the HDRC camera. A notchfilter can be used to block significant radiation from the sun, forexample. This notch filter can be made as a part of the lens through theplacement of various coatings onto the lens surface.

[0170] Liquid crystals operate rapidly and give as much as a dynamicrange of 10,000 to 1 but may create a pixel interference affect.Electrochromic glass operates more slowly but more uniformly therebyeliminating the pixel affect. The pixel effect arises whenever there isone pixel device in front of another. This results in various aliasing,Moire patterns and other ambiguities. One way of avoiding this is toblur the image. Another solution is to use a large number of pixels andcombine groups of pixels to form one pixel of information and thereby toblur the edges to eliminate some of the problems with aliasing and Moirepatterns.

[0171] One straightforward approach is the use of a mechanical iris.Standard cameras already have response times of several tens ofmilliseconds range. They will switch, for example, in a few frames on atypical video camera (1 frame=0.033 seconds). This is sufficiently fastfor categorization but much too slow for dynamic out-of-positiontracking.

[0172] An important feature of the IMS Chips HDRC camera is that thefull dynamic range is available at each pixel. Thus, if there aresignificant variations in the intensity of light within the vehicle, andthereby from pixel to pixel, such as would happen when sunlight streamsand through a window, the camera can automatically adjust and providethe optimum exposure on a pixel by pixel basis. The use of the camerahaving this characteristic is very beneficial to the invention describedherein and contributes significantly to system accuracy. CCDs have arather limited dynamic range due to their inherent linear response andconsequently cannot come close to matching the performance of humaneyes. A key advantage of the IMS Chips HDRC camera is its logarithmicresponse which comes closest to matching that of the human eye.

[0173] Another approach, which is applicable in some vehicles, is torecord an image without the infrared illumination and then a secondimage with the infrared illumination and to then subtract the firstimage from the second image. In this manner, illumination caused bynatural sources such as sunlight or even from light bulbs within thevehicle can be subtracted out. Naturally, using the logarithmic pixelsystem of the IMS Chips camera care must be taken to include thelogarithmic effect during the subtraction process. For some cases,natural illumination such as from the sun, light bulbs within thevehicle, or radiation emitted by the object itself can be used alonewithout the addition of a special source of infrared illumination.

[0174] Other imaging systems such as CCD arrays can also of course beused with this invention. However, the techniques will be quitedifferent since the camera is very likely to saturate when bright lightis present and to require the full resolution capability when the lightis dim. Generally when practicing this invention the interior of thepassenger compartment will be illuminated with infrared radiation.

[0175] There are other bright sources of infrared that must be accountedfor. These include the sun and any light bulbs that may be presentinside the vehicle. This lack of a high dynamic range inherent with theCCD technology requires the use of an iris, liquid crystal, orelectrochromic glass filter to be placed between the camera and thescene. Even with these filters however, some saturation will take placewith CCD cameras under bright sun or incandescent lamp exposure. Thissaturation reduces the accuracy of the image and therefore the accuracyof the system. In particular the training regimen that must be practicedwith CCD cameras is more severe since all of the saturation cases mustbe considered since the camera is unable to appropriately adjust. Thus,although CCD cameras can be use, HDRC logarithmic cameras such asmanufactured by IMS Chips are preferred. They not only provide asignificantly more accurate image but also significantly reduce theamount of training effort and associated data collection that must beundertaken during the development of the neural network algorithm orother computational intelligence system. In some applications, it ispossible to use other more deterministic image processing or patternrecognition systems than neural networks.

[0176] Another very important feature of the HDRC camera from IMS Chipsis that the shutter time is constant at less than 100 ns irrespective ofbrightness of the scene. The pixel data arrives at constant ratesynchronous with the internal imager clock. Random access to each pixelfacilitates high-speed intelligent access to any sub-frame (block) sizeor sub-sampling ratio and a trade-off of frame speed and frame sizetherefore results. For example, a scene with 128 K pixels per frame canbe taken at 120 frames per second, or about 8 milliseconds per frame,whereas a sub-frame can be taken in run at as high as 4000 frames persecond with 4 K pixels per frame. This combination allows the maximumresolution for the identification and classification part of theoccupant sensor problem while permitting a concentration on thoseparticular pixels which track the head or chest, as described above, fordynamic out-of-position tracking. In fact the random access features ofthese cameras can be used to track multiple parts of the imagesimultaneously while ignoring the majority of the image, and do so atvery high speed. For example, the head can be tracked simultaneouslywith the chest by defining two separate sub-frames that need not beconnected. This random access pixel capability, therefore, is optimallysuited or recognizing and tracking vehicle occupants. It is also suitedfor monitoring the environment outside of the vehicle for purposes ofblind spot detection, collision avoidance and anticipatory sensing.Photobit Corporation of 135 North Los Robles Ave., Suite 700, Pasadena,Calif. 91101 manufactures another camera with some characteristicssimilar to the IMS Chips camera. Other competitive cameras can beexpected to appear on the market.

[0177] Photobit refers to their Active Pixel Technology as APS.According to Photobit, in the APS, both the photodetector and readoutamplifier are part of each pixel. This allows the integrated charge tobe converted into a voltage in the pixel that can then be read out overX-Y wires instead of using a charge domain shift register as in CCDs.This column and row addressability (similar to common DRAM) allows forwindow of interest readout (windowing) which can be utilized for on chipelectronic pan/tilt and zoom. Windowing provides added flexibility inapplications, such as disclosed herein, needing image compression,motion detection or target tracking. The APS utilizes intra-pixelamplification in conjunction with both temporal and fixed pattern noisesuppression circuitry (i.e. correlated double sampling), which producesexceptional imagery in terms of wide dynamic range (˜75 dB) and lownoise (˜15 e-rms noise floor) with low fixed pattern noise (<0.15% sat).Unlike CCDs, the APS is not prone to column streaking due to bloomingpixels. This is because CCDs rely on charge domain shift registers thatcan leak charge to adjacent pixels when the CCD registers overflows.Thus, bright lights “bloom” and cause unwanted streaks in the image. Theactive pixel can drive column busses at much greater rates than passivepixel sensors and CCDs. On-chip analog-to-digital conversion (ADC)facilitates driving high speed signals off chip. In addition, digitaloutput is less sensitive to pickup and crosstalk, facilitating computerand digital controller interfacing while increasing system robustness. Ahigh speed APS recently developed for a custom binary output applicationproduced over 8,000 frames per second, at a resolution of 128×128pixels. It is possible to extend this design to a 1024×1024 array sizeand achieve greater than 1000 frames per second for machine vision. Allof these features are important to many applications of this invention.

[0178] These advanced cameras, as represented by the HDRC and the APScameras, now make it possible to more accurately monitor the environmentin the vicinity of the vehicle. Heretofore, the large dynamic range ofenvironmental light has either blinded the cameras when exposed tobright light or else made them unable to record images when the lightlevel was low. Even the HDRC camera with its 120 dB dynamic range may bemarginally sufficient to handle the fluctuations in environmental lightthat occur. Thus, the addition of a electrochromic, liquid crystal, orother similar filter may be necessary. This is particularly true forcameras such as the Photobit APS camera with its 75 dynamic range.

[0179] At about 120 frames per second, these cameras are adequate forcases where the relative velocity between vehicles is low. There aremany cases, however, where this is not the case and a much highermonitoring rate is required. This occurs for example, in collisionavoidance and anticipatory sensor applications. The HDRC camera isoptimally suited for handling these cases since the number of pixelsthat are being monitored can be controlled resulting in a frame rate ashigh as about 4000 frames per second with a smaller number of pixels.

[0180] Another key advantage of the HDRC camera is that it is quitesensitive to infrared radiation in the 0.8 to 1 manometer wavelengthrange. This range is generally beyond visual range for humans permittingthis camera to be used with illumination sources that are not visible tothe human eye. Naturally, a notch filter is frequently used with thecamera to eliminate unwanted wavelengths. These cameras are availablefrom the Institute for Microelectronics (IMS Chips), Allamndring 30a,D-70569 Stuttgart, Germany with a variety of resolutions ranging from512 by 256 to 720 by 576 pixels and can be custom fabricated for theresolution and response time required.

[0181] An optical infrared transmitter and receiver assembly is showngenerally at 112 in FIG. 1B and is mounted onto the instrument panelfacing the windshield. Assembly 112 can either be recessed below theupper face of the instrument panel or mounted onto the upper face of theinstrument panel. Assembly 112, shown enlarged, comprises a source ofinfrared radiation, or another form of electromagnetic radiation, and aCCD or CMOS array of typically 160 pixels by 160 pixels. In thisembodiment, the windshield is used to reflect the illumination lightprovided by the infrared radiation toward the objects in the passengercompartment and also reflect the light being reflected back by theobjects in the passenger compartment, in a manner similar to the“heads-up” display which is now being offered on several automobilemodels. The “heads-up” display, of course, is currently used only todisplay information to the driver and is not used to reflect light fromthe driver to a receiver. Once again, unless one of the distancemeasuring systems as described below is used, this system alone cannotbe used to determine distances from the objects to the sensor. Its mainpurpose is object identification and monitoring. Depending on theapplication, separate systems can be used for the driver and for thepassenger. In some cases, the cameras located in the instrument panelwhich receive light reflected off of the windshield can be co-locatedwith multiple lenses whereby the respective lenses aimed at the driverand passenger seats respectively.

[0182] Assembly 112 is actually about two centimeters in diameter and isshown greatly enlarged in FIG. 1B. Also, the reflection area on thewindshield is considerably smaller than illustrated and specialprovisions are made to assure that this area of the windshield is flatand reflective as is done generally when heads-up displays are used. Forcases where there is some curvature in the windshield, it can be atleast partially compensated for by the CCD optics.

[0183] When using the surface of the windshield as a reflector ofinfrared radiation, care must be taken to assure that the desiredreflectivity at the frequency of interest is achieved. Mirror materials,such as metals and other special materials manufactured by EastmanKodak, have a reflectivity for infrared frequencies that issubstantially higher than at visible frequencies. They are thuscandidates for coatings to be placed on the windshield services for thispurpose. If two spaced apart CCD arrays are used, then the distance tothe various objects within the passenger compartment can be found byusing a triangulation algorithm which locates similar features on bothimages and determines their relative location on the images. Analternate method is to use a lens with a short focal length. In thiscase, the lens is mechanically focused, e.g., automatically, directly orindirectly, by the control circuitry 120, to determine the clearestimage and thereby obtain the distance to the object. This is similar tocertain camera auto-focusing systems such as one manufactured by Fuji ofJapan. Naturally, other methods can be used as described in the patentsand patent applications referenced above.

[0184] Instead of focusing the lens, the lens could be moved relative tothe array to thereby adjust the image on the array. Instead of movingthe lens, the array could be moved to achieve the proper focus. Inaddition, it is also conceivable that software could be used to focusthe image without moving the lens or the array.

[0185] Once a vehicle interior monitoring system employing asophisticated pattern recognition system, such as a neural network, isin place, it is possible to monitor the motions of the driver over time,and his/her response to various stimuli, and determine if he or she isfalling asleep, has become incapacitated or otherwise unable to operatethe vehicle. In such an event, the vehicle can be caused to respond in anumber of different ways. One such system is illustrated in FIG. 1B andconsists of a monitoring system having the transducer assembly 112coupled to a microprocessor in control circuitry 120 which is programmedto compare the motions of the driver over time and trained to recognizechanges in behavior representative of becoming incapacitated, e.g., theeyes blinking erratically and remaining closed for ever longer periodsof time. If the system determines that there is a reasonable probabilitythat the driver has fallen asleep, for example, then it can activate analarm, e.g., turn on a warning light shown here as 124 or send a warningsound. If the driver fails to respond to the warning by pushing a button122, for example, then the horn and lights of the vehicle can beoperated in a manner to warn other vehicles and the vehicle may bebrought to a stop. Naturally, other responses can also be programmed andother tests of driver attentiveness can be used without resorting toattempting to monitor the motions of the driver's eyes.

[0186] The use of the windshield as a reflector is particularly usefulwhen monitoring the eyes of the driver. The reflections from the corneaare highly directional as every driver knows whose lights have reflectedoff the eyes of an animal on the roadway. For this to be effective, theeyes of the driver must be looking at the radiation source. Since thedriver is presumably looking through the windshield, the source of theradiation must also come from the windshield and the reflections fromthe driver's eyes must also be in the direction of the windshield. Usingthis technique, the time that the driver spends looking through thewindshield can be monitored and if that time drops below some thresholdvalue it can be presumed that the driver is not attentive and may besleeping or otherwise incapacitated.

[0187] An even more sophisticated system of monitoring the behavior ofthe driver is to track the driver's eye motions using such techniques asare described in: Freidman et al., U.S. Pat. No. 4,648,052 entitled “EyeTracker Communication System”; Heyner et al., U.S. Pat. No. 4,720,189entitled “Eye Position Sensor”; Hutchinson, U.S. Pat. No. 4,836,670entitled “Eye Movement Detector”; and Hutchinson, U.S. Pat. No.4,950,069 entitled “Eye Movement Detector With Improved Calibration andSpeed”, all of which are incorporated herein by reference as well asU.S. Pat. Nos. 5,008,946 and 5,305,012 referenced above. The detectionof the impaired driver in particular can be best determined by thesetechniques. These systems make use of pattern recognition techniquesplus, in many cases, the transmitter and CCD receivers must beappropriately located so that the reflection off of the cornea of thedriver's eyes can be detected as discussed in the above referencedpatents. The size of the CCD arrays used herein permits their location,sometimes in conjunction with a reflective windshield, where thiscorneal reflection can be detected with some difficulty. Sunglasses orother items can interfere with this process.

[0188] The location of the eyes of the driver, for this application, isgreatly facilitated by the teachings of this invention as describedabove. Although others have suggested the use of eye motions and cornealreflections for drowsiness determination, up until now there has notbeen a practical method for locating the driver's eyes with sufficientprecision and reliability as to render this technique practical. Also,although sunglasses might defeat such a system, most drowsiness causedaccidents happen at night where it is less likely that sunglasses areworn.

[0189] The eye tracker systems discussed above are facilitated by theinstant invention since one of the main purposes of determining thelocation of the driver's eyes either by directly locating them withtrained pattern recognition technology or by inferring their locationfrom the location of the driver's head, is so that the seat can beautomatically positioned to place the driver's eyes into the“eye-ellipse”. The eye-ellipse is the proper location for the driver'seyes to permit optimal operation of the vehicle and for the location ofthe mirrors etc. Thus, if the location of the driver's eyes are known,then the driver can be positioned so that his or her eyes are preciselysituated in the eye ellipse and the reflection off of the eye can bemonitored with a small eye tracker system. Also, by ascertaining thelocation of the driver's eyes, a rear view mirror positioning device canbe controlled to adjust the mirror 105 to an optimal position.

[0190] In addition to finding the location of the eyes, the location ofthe ears is becoming more important. Many automobile accidents are nowbeing caused by driver's holding on and talking into cellular phones.Vehicle noise significantly deteriorates the quality of the sound heardby the driver from speakers. This problem can be solved through the useof hypersound and by knowing the location of the ears of the driver.Hypersound permits the precise focusing of sound waves along a line fromthe speaker with little divergence of the sound field. Thus, if thelocations of the ears of the driver are known, the sound can beprojected to them directly thereby overcoming much of the vehicle noise.In addition to the use of hypersound, directional microphones are wellknown in the microphone art which are very sensitive to sound comingfrom a particular direction. If the driver has been positioned so thathis eyes are in the eye ellipse, then the location of the driver's mouthis also accurately known and a fixed position directional microphone canbe used to selectively sense sound emanating from the mouth of thedriver. In many cases, the sensitivity of the microphone can be designedto include a large enough area such that most motions of the driver'shead can be tolerated. Alternately the direction of the microphone canbe adjusted using motors or the like. Systems of noise cancellation nowalso become possible if the ear locations are precisely known and noisecanceling microphones as described in U.S. provisional patentapplication Serial No. 60/110,973, which is incorporated herein byreference, if the location of the driver's mouth is known.

[0191] Infrared waves are shown coming from the front and backtransducer assemblies 110 and 113 in FIG. 1C.

[0192]FIG. 1D illustrates two optical systems each having a source ofinfrared radiation and a CCD or CMOS array receiver. The price of sucharrays has dropped dramatically recently making them practical forinterior and exterior vehicle monitoring. In this embodiment,transducers 110 and 113 are CMOS arrays having 160 pixels by 160 pixelscovered by a lens. In some applications, this can create a “fisheye”effect whereby light from a wide variety of directions can be captured.One such transducer placed by the dome light or other central positionin the vehicle headliner, such as the transducer designated 113, canmonitor the entire vehicle interior with sufficient resolution todetermine the occupancy of the vehicle, for example. CCD's such as thoseused herein are available from Marshall Electronics Inc. of Culver City,Calif. A fisheye lens is “ . . . a wide-angle photographic lens thatcovers an angle of about 180°, producing a circular image withexaggerated foreshortening in the center and increasing distortiontoward the periphery”. (The American Heritage Dictionary of the EnglishLanguage, Third Edition, 1992 by Houghton Mifflin Company). Thisdistortion of a fisheye lens can be substantially changed by modifyingthe shape of the lens to permit particular portions of the interiorpassenger compartment to be observed. Also, in many cases the full 180°is not desirable and a lens which captures a smaller angle may be used.Although primarily spherical lenses are illustrated herein, it isunderstood that the particular lens design will depend on the locationin the vehicle and the purpose of the particular receiver.

[0193] CCD arrays are in common use in television cameras, for example,to convert an image into an electrical signal. For the purposes herein,a CCD will be defined to include all devices, including CMOS arrays, APSarrays, artificial retinas and particularly HDRC arrays, which arecapable of converting light frequencies, including infrared, visible andultraviolet, into electrical signals. The particular CCD array used formany of the applications disclosed herein is implemented on a singlechip that is less than two centimeters on a side. Data from the CCDarray is digitized and sent serially to an electronic circuit (at timesdesignated 120 herein) containing a microprocessor for analysis of thedigitized data. In order to minimize the amount of data that needs to bestored, initial processing of the image data takes place as it is beingreceived from the CCD array, as discussed in more detail above. In somecases, some image processing can take place on the chip such asdescribed in the Kage et al. artificial retina article referenced above.

[0194] One method of determining distance to an object directly withoutresorting to range finders, which require multiple arrays, is to use amechanical focusing system. However, the use of such an apparatus iscumbersome, expensive, and slow and has questionable reliability. Analternative is to use the focusing systems described in the abovereferenced U.S. Pat. Nos. 5,193,124 and 5,003,166, however, such systemsrequire expensive hardware and/or elaborate algorithms. Anotheralternative is illustrated in FIG. 1D where transducer 116 is aninfrared source having a wide transmission angle such that the entirecontents of the front driver's seat is illuminated. Receiving CCDtransducers 117 and 118 are shown spaced apart so that a stereographicanalysis can be made by the control circuitry 120. This circuitry 120contains a microprocessor with appropriate pattern recognitionalgorithms along with other circuitry as described above. In this case,the desired feature to be located is first selected from one of the tworeturned images from either CCD transducer 117 or 118. The software thendetermines the location of the same feature, through correlationanalysis or other methods, on the other image and thereby, throughanalysis familiar to those skilled in the art, determines the distanceof the feature from the transducers.

[0195] Transducers 116-118 are illustrated mounted onto the A-pillar ofthe vehicle, however, since these transducers are quite small, typicallyapproximately 2 cm on a side, they could alternately be mounted onto thewindshield itself, or other convenient location which provides a clearview of the portion of the passenger compartment being monitored. Otherpreferred mounting locations include the headliner above and also theside of the seat.

[0196] With respect to the connection between the transducers 110-114and 116-118 and the control circuitry 120, a portion of this connectionis shown as wires. It should be understood that all of the connectionsbetween the transducers 110-114 and 116-118 and the control circuitry120 may be wires, either individual wires leading from the controlcircuitry 120 to each of the transducers 110-114 and 116-118 or one ormore wire buses.

[0197] With respect to the position of the control circuitry 120 in thedashboard of the vehicle, this position is for illustration purposesonly and does not limit the location of the control circuitry 120.Rather, the control circuitry 120 may be located anywhere convenient ordesired in the vehicle.

[0198] A new class of laser range finders has particular applicationhere. This product, as manufactured by Power Spectra, Inc. of Sunnyvale,Calif., is a GaAs pulsed laser device which can measure up to 30 meterswith an accuracy of <2 cm and a resolution of <1 cm. This system isimplemented in combination with transducer 116 and one of the receivingtransducers 117 or 118 may thereby be eliminated. Once a particularfeature of an occupying item of the passenger compartment has beenlocated, this device is used in conjunction with an appropriate aimingmechanism to direct the laser beam to that particular feature. Thedistance to that feature is then known to within 2 cm and withcalibration even more accurately. In addition to measurements within thepassenger compartment, this device has particular applicability inanticipatory sensing and blind spot monitoring applications exterior tothe vehicle. An alternate technology using range gating to measure thetime of flight of electromagnetic pulses with even better resolution canbe developed based on the teaching of the McEwan patents listed aboveand incorporated herein by reference.

[0199] A more accurate acoustic system for determining the distance to aparticular object, or a part thereof, in the passenger compartment isexemplified by transducers 111A in FIG. 1E. In this case, threeultrasonic transmitter/receivers are shown spaced apart mounted onto theA-pillar of the vehicle. The A-pillar is the forward most roof supportpillar and also supports the windshield. Due to the wavelength, it isdifficult to get a narrow beam using ultrasonics without either usinghigh frequencies that have limited range or a large transducer. Acommonly available 40 kHz transducer, for example, is about 1 cm. indiameter and emits a sonic wave that spreads at about a sixty-degreeangle. To reduce this angle requires making the transducer larger indiameter. An alternate solution is to use several transducers and tophase the transmissions so that they arrive at the intended part of thetarget in phase. Reflections from the selected part of the target arethen reinforced whereas reflections from adjacent parts encounterinterference with the result that the distance to the brightest portionwithin the vicinity of interest can be determined. By varying the phaseof transmission from the three transducers 111A, the location of areflection source on a curved line can be determined. In order to locatethe reflection source in space, at least one additionaltransmitter/receiver is required which is not co-linear with the others.The accuracy of the measurement can be determined by those skilled inthe art of phased array radar as the relevant equations are applicablehere. The waves shown in FIG. 1E coming from the three transducers 111Aare actually only the portions of the waves which arrive at the desiredpoint in space together in phase. The effective direction of these wavestreams can be varied by changing the transmission phase between thethree transmitters. A determination of the approximate location of apoint of interest on the occupant is accomplished by the CCD array andappropriate analysis and the phasing of the ultrasonic transmitters isdetermined so that the distance to the desired point can be determined.

[0200]FIG. 2A is a side view, with certain portions removed or cut away,of a portion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors(transmitter/receiver assemblies or transducers) 110, 111A, 113, 114,210, 211A, 213, 214, and 224. Each of these sensors is illustrated ashaving a lens and is shown enlarged in size for clarity. In a typicalactual device, the diameter of the lens is approximately 2 cm and itprotrudes from the mounting surface by approximately 1 cm. This smallsize renders these devices almost unnoticeable by vehicle occupants.Since these sensors are optical, it is important that the lens surfaceremains relatively clean. Control circuitry 120, which is coupled toeach transducer, contains a self-diagnostic feature where the imagereturned by a transducer is compared with a stored image and theexistence of certain key features is verified. If a receiver fails thistest, a warning is displayed to the driver which indicates that cleaningof the lens surface is required. The technology illustrated in FIG. 2Acan be used for numerous purposes relating to monitoring of the space inthe passenger compartment behind the driver including: (i) thedetermination of the presence and position of objects in the rearseat(s), (ii) the determination of the presence, position andorientation of child seats 230 in the rear seat, (iii) the monitoring ofthe rear of an occupant's head 242, (iv) the monitoring of the positionof occupant 240, (v) the monitoring of the position of the occupant'sknees 241, (vi) the monitoring of the occupant's position relative tothe airbag 250, (vii) the measurement of the occupant's height, as wellas other monitoring functions as described elsewhere herein.

[0201] Information relating to the space behind the driver can beobtained by processing the data obtained by the sensors 210,211A,213 and214, which data would be in the form of images if optical sensors areused as in the preferred embodiment. Such information can be thepresence of a particular occupying item or occupant, e.g., a rear facingchild seat 230 as shown in FIG. 2A, as well as the location or positionof occupying items. Additional information obtained by the opticalsensors can include an identification of the occupying item. Theinformation obtained by the control circuitry by processing theinformation from sensors 210,211A,213 and 214 may be used to affect anyother system in the vehicle in a similar manner as the information fromthe sensors which monitor the front seat is used as described herein,such as the airbag system. Processing of the images obtained by thesensors to determine the presence, position and/or identification of anyoccupants or occupying item can be effected using a pattern recognitionalgorithm in any of the ways discussed herein, e.g., a trained neuralnetwork.

[0202] Sensors 210,211A,213,214 can also be microwave radar sensorswhich transmit and receive radar waves. As such, it is possible todetermine the presence of an object in the rear seat and the distancebetween the object and the sensors. Using multiple radar sensors, itwould be possible to determine the contour of an object in the rear seatand thus using pattern recognition techniques, the classification oridentity of the object. Motion of objects in the rear seat can also bedetermined using radar sensors. For example, if the radar sensors aredirected toward a particular area and/or are provided with the abilityto detect motion in a predetermined frequency range, they can be used todetermine the presence of children or pets left in the vehicle, i.e., bydetecting heartbeats or other body motions such as movement of the chestcavity.

[0203]FIG. 2B is a perspective view corresponding to the embodimentshown in FIG. 2A illustrating some of the transducer mounting locations(including sensor 110A). The passenger side airbag module is designated104A and is mounted in the dashboard or instrument panel of the vehicle.

[0204] The occupant position sensor in any of its various forms isintegrated into the airbag system circuitry as shown schematically inFIG. 3. In this example, the occupant position sensors are used as aninput to a smart electronic sensor and diagnostic system. The electronicsensor determines whether one or more of the airbags should be deployedbased on the vehicle acceleration crash pulse, or crush zone mountedcrash sensors, or a combination thereof, and the occupant positionsensor determines whether the occupant is too close to any of theairbags and therefore that the deployment should not take place. In FIG.3, the electronic crash sensor located within the sensor and diagnosticunit determines whether the crash is of such severity as to requiredeployment of one or more of the airbags. The occupant position sensorsdetermine the location of the vehicle occupants relative to the airbagsand provide this information to the sensor and diagnostic unit that thendetermines whether it is safe to deploy each airbag and/or whether thedeployment parameters should be adjusted. The arming sensor, if one ispresent, also determines whether there is a vehicle crash occurring. Insuch a case, if the sensor and diagnostic unit and the arming sensorboth determine that the vehicle is undergoing a crash requiring one ormore airbags and the position sensors determine that the occupants aresafely away from the airbag(s), the airbag(s), or inflatable restraintsystem, is deployed.

[0205] A particular implementation of an occupant position sensor havinga range of from 0 to 2 meters (corresponding to an occupant position offrom 0 to 1 meter since the signal must travel both to and from theoccupant) using infrared is illustrated in the block diagram schematicof FIG. 4. The operation is as follows. A 48 MHz signal, f1, isgenerated by a crystal oscillator 401 and fed into a frequency tripler402 which produces an output signal at 144 MHz. The 144 MHz signal isthen fed into an infrared diode driver 403 which drives the infrareddiode 404 causing it to emit infrared light modulated at 144 MHz and areference phase angle of zero degrees. The infrared diode 404 isdirected at the vehicle occupant. A second signal f2 having a frequencyof 48.05 MHz, which is slightly greater than f1, is similarly fed from acrystal oscillator 405 into a frequency tripler 406 to create afrequency of 144.15 MHz. This signal is then fed into a mixer 407 whichcombines it with the 144 MHz signal from frequency tripler 402. Thecombined signal from the mixer 407 is then fed to filter 408 whichremoves all signals except for the difference, or beat frequency,between 3 times f1 and 3 times f2, of 150 kHz. The infrared signal whichis reflected from the occupant is received by receiver 409 and fed intopre-amplifier 411, a resistor 410 to bias being coupled to theconnection between the receiver 409 and the pre-amplifier 411. Thissignal has the same modulation frequency, 144 MHz, as the transmittedsignal but now is out of phase with the transmitted signal by an angle xdue to the path that the signal took from the transmitter to theoccupant and back to the receiver. The output from pre-amplifier 411 isfed to a second mixer 412 along with the 144.15 MHz signal from thefrequency tripler 406. The output from mixer 412 is then amplified by anautomatic gain amplifier 413 and fed into filter 414. The filter 414eliminates all frequencies except for the 150 kHz difference, or beat,frequency, in a similar manner as was done by filter 408. The resulting150 kHz frequency, however, now has a phase angle x relative to thesignal from filter 408. Both 150 kHz signals are now fed into a phasedetector 415 which determines the magnitude of the phase angle x. It canbe shown mathematically that, with the above values, the distance fromthe transmitting diode to the occupant is x/345.6 where x is measured indegrees and the distance in meters. The velocity can also be obtainedusing the distance measurement as represented by 416. An alternatemethod of obtaining distance information, as discussed above, is to usethe teachings of the McEwan patents discussed above.

[0206] Although the embodiment in FIG. 4 uses infrared, it is possibleto use other frequencies of energy without deviating from the scope ofthe invention.

[0207] The applications described herein have been illustrated using thedriver of the vehicle. The same systems of determining the position ofthe occupant relative to the airbag apply to front and rear seatedpassengers, sometimes requiring minor modifications. It is likely thatthe sensor required triggering time based on the position of theoccupant will be different for the driver than for the passenger.Current systems are based primarily on the driver with the result thatthe probability of injury to the passenger is necessarily increasedeither by deploying the airbag too late or by failing to deploy theairbag when the position of the driver would not warrant it but thepassenger's position would. With the use of occupant position sensorsfor both the passenger and driver, the airbag system can be individuallyoptimized for each occupant and result in further significant injuryreduction. In particular, either the driver or passenger system can bedisabled if either the driver or passenger is out-of-position or if thepassenger seat is unoccupied.

[0208] There is almost always a driver present in vehicles that areinvolved in accidents where an airbag is needed. Only about 30% of thesevehicles, however, have a passenger. If the passenger is not present,there is usually no need to deploy the passenger side airbag. Theoccupant monitoring system, when used for the passenger side with properpattern recognition circuitry, can also ascertain whether or not theseat is occupied, and if not, can disable the deployment of thepassenger side airbag and thereby save the cost of its replacement. Thesame strategy applies also for monitoring the rear seat of the vehicle.Also, a trainable pattern recognition system, as used herein, candistinguish between an occupant and a bag of groceries, for example.Finally, there has been much written about the out-of-position child whois standing or otherwise positioned adjacent to the airbag, perhaps dueto pre-crash braking. The occupant position sensor described herein canprevent the deployment of the airbag in this situation as well as in thesituation of a rear facing child seat as described above.

[0209] The use of trainable pattern recognition technologies such asneural networks is an important part of the instant invention, althoughother non-trained pattern recognition systems such as fuzzy logic,correlation, Kalman filters, and sensor fusion (a derivative of fuzzylogic) can also be used. These technologies are implemented usingcomputer programs to analyze the patterns of examples to determine thedifferences between different categories of objects. These computerprograms are derived using a set of representative data collected duringthe training phase, called the training set. After training, thecomputer programs output a computer algorithm containing the rulespermitting classification of the objects of interest based on the dataobtained after installation in the vehicle. These rules, in the form ofan algorithm, are implemented in the system that is mounted onto thevehicle. The determination of these rules is central to the patternrecognition techniques used in this invention. Artificial neuralnetworks using back propagation are thus far the most successful of therule determination approaches, however, research is underway to developsystems with many of the advantages of back propagation neural networks,such as learning by training, without the disadvantages, such as theinability to understand the network and the possibility of notconverging to the best solution. In particular, back propagation neuralnetworks will frequently give an unreasonable response when presentedwith data than is not within the training data. It is well known thatneural networks are good at interpolation but poor at extrapolation. Acombined neural network fuzzy logic system, on the other hand, cansubstantially solve this problem. Additionally, there are many otherneural network systems in addition to back propagation. In fact, onetype of neural network may be optimum for identifying the contents ofthe passenger compartment and another for determining the location ofthe object dynamically.

[0210] In some implementations of this invention, such as thedetermination that there is an object in the path of a closing window asdescribed below, the rules are sufficiently obvious that a trainedresearcher can look at the returned optical signals and devise analgorithm to make the required determinations. In others, such as thedetermination of the presence of a rear facing child seat or anoccupant, artificial neural networks are frequently used to determinethe rules. One such set of neural network software for determining thepattern recognition rules, is available from the NeuralWare Corporationof Pittsburgh, Pa. Numerous books and articles, including more that 500U.S. patents, describe neural networks in great detail and thus thetheory and application of this technology is well known and will not berepeated here. Except in a few isolated situations where neural networkshave been used to solve particular problems limited to engine control,for example, they have not heretofore been applied to automobiles andtrucks.

[0211] The system generally used in the instant invention, therefore,for the determination of the presence of a rear facing child seat, anoccupant, or an empty seat is the artificial neural network or aneural-fuzzy system. In this case, the network operates on the returnedsignals from the CCD array as sensed by transducers 110, 111, 113 and114 (not shown) in FIG. 5, for example. For the case of the frontpassenger seat, for example, through a training session, the system istaught to differentiate between the three cases. This is done byconducting a large number of experiments where available child seats areplaced in numerous positions and orientations on the front passengerseat of the vehicle. Similarly, a sufficiently large number ofexperiments are run with human occupants and with boxes, bags ofgroceries and other objects. As many as 1,000,000 such experiments arerun before the neural network is sufficiently trained so that it candifferentiate among the three cases and output the correct decision witha very high probability.

[0212] Once the network is determined, it is possible to examine theresult to determine, from the algorithm created by the NeuralWaresoftware, the rules that were finally arrived at by the trial and errortraining technique. In that case, the rules can then be programmed intoa microprocessor. Alternately, a neural computer can be used toimplement the net directly. In either case, the implementation can becarried out by those skilled in the art of pattern recognition usingneural networks. If a microprocessor is used, a memory device is alsorequired to store the data from the analog to digital converters whichdigitize the data from the receiving transducers. On the other hand, ifa neural network computer is used, the analog signal can be fed directlyfrom the transducers to the neural network input nodes and anintermediate memory is not required. Memory of some type is needed tostore the computer programs in the case of the microprocessor system andif the neural computer is used for more than one task, a memory isneeded to store the network specific values associated with each task.

[0213] There are several methods measuring the height of the driver foruse in automatically adjusting the seat or for adjusting the seatbeltanchorage point. Some alternatives are shown in FIG. 5, which is a sideplan view of the front portion of the passenger compartment showingthree height measuring transducers or sensors 110,111,113, all of whichare mounted on or near the headliner. These transducers may already bepresent because of other implementations of the vehicle interioridentification and monitoring system described herein. The combinationof four transducers can determine, by the methods described above, thelocation of the head with great accuracy.

[0214] Optical transducers using CCD arrays are now becoming pricecompetitive and, as mentioned above, will soon be the technology ofchoice for interior vehicle monitoring. A single CCD array of 160 by 160pixels, for example, coupled with the appropriate trained patternrecognition software, can be used to form an image of the head of anoccupant and accurately locate the head for some of the purposes of thisinvention.

[0215] The position of the rear of the head can also be known once thelocus of the head has been determined. This information can be used todetermine the distance from the headrest to the rearmost position of theoccupant's head and to control the position of the headrest so that itis properly positioned behind the occupant's head to offer optimumsupport in the event of a rear impact. Although the headrest of mostvehicles is adjustable, it is rare for an occupant to position itproperly, if at all. Each year, there are in excess of 400,000 whiplashinjuries in vehicle impacts approximately 90,000 of which are from rearimpacts (source: National Highway Traffic Safety Administration,(NHTSA)). A properly positioned headrest could substantially reduce thefrequency of such injuries that can be accomplished by the head detectorof this invention. The head detector is connected to the headrestcontrol mechanism and circuitry 540. This mechanism is capable of movingthe headrest up and down and, in some cases, rotating it fore and aft.Thus, the control circuitry 120 may be coupled to headrest controlmechanism and circuitry 540 to adjust the headrest based on thedetermined location of the rear of the occupant's head.

[0216] An occupant position sensor for side impacts used with a doormounted airbag system is illustrated at 530 in FIG. 5. This sensor hasthe particular task of monitoring the space adjacent to the door-mountedairbag. Sensor 530 may also be coupled to control circuitry 120 whichcan process and use the information provided by sensor 530 in thedetermination of the location or identity of the occupant or location ofa part of the occupant.

[0217] Seatbelts are most effective when the upper attachment point tothe vehicle is positioned vertically close to the shoulder of theoccupant being restrained. If the attachment point is too low, theoccupant experiences discomfort from the rubbing of the belt on his orher shoulder. If it is too high, the occupant may experience discomfortdue to the rubbing of the belt against his or her neck and the occupantwill move forward by a greater amount during a crash which may result inhis or her head striking the steering wheel. For these reasons, it isdesirable to have the upper seatbelt attachment point located slightlyabove the occupant's shoulder. To accomplish this for various sizedoccupants, the location of the occupant's shoulder must be known, whichcan be accomplished by the vehicle interior monitoring system describedherein.

[0218] Such a system is illustrated in FIG. 6, which is a side planerview of a seatbelt anchorage adjustment system. In this system, infraredtransmitter and CCD array receivers 620 and 621 are positioned in aconvenient location proximate the occupant's shoulder, such as inconnection with the headliner, above and usually to the outside of theoccupant's shoulder. An appropriate pattern recognition system, as maybe resident in control circuitry 120 to which the receivers 620,621 arecoupled, as described above is then used to determine the location andposition of the shoulder. This information is provided by controlcircuitry 120 to the seatbelt anchorage height adjustment system 632(through a conventional coupling arrangement), shown schematically,which moves the attachment point 631 of the seatbelt 630 to the optimumvertical location for the proper placement of the seatbelt 630.

[0219]FIG. 7 is an angular perspective overhead view of a vehicle 710about to be impacted in the side by an approaching vehicle 720, wherevehicle 710 is equipped with an anticipatory sensor system showing atransmitter 730 transmitting electromagnetic, such as infrared, wavestoward vehicle 720. This is one example of many of the uses of theinstant invention for exterior monitoring. The transmitter 730 isconnected to an electronic module 740. Module 740 contains circuitry 742to drive transmitter 730 and circuitry 744 to process the returnedsignals from receivers 734 and 736 which are also coupled to module 740.Circuitry 744 contains a processor such as a neural computer 745, whichperforms the pattern recognition determination based on signals fromreceivers 734 and 736 (FIG. 7A). Receivers 734 and 736 are mounted ontothe B-Pillar of the vehicle and are covered with a protectivetransparent cover. An alternate mounting location is shown as 738 whichis in the door window trim panel where the rear view mirror (not shown)is frequently attached. One additional advantage of this system is theability of infrared to penetrate fog and snow better than visible lightwhich makes this technology particularly applicable for blind spotdetection and anticipatory sensing applications. Although it is wellknown that infrared can be significantly attenuated by both fog andsnow, it is less so than visual light depending on the frequency chosen.(See for example L. A. Klein, Millimeter-Wave and Infrared MultisensorDesign and Signal Processing, Artech House, Inc, Boston 1997, ISBN0-89006-764-3 which is incorporated herein by reference).

[0220] The same system can also be used for the detection of objects inthe blind spot of the vehicle and the image or an icon representativethereof displayed for the operator to see (display 748 discussed belowwith reference to FIG. 15), or a warning system activated, if theoperator attempts to change lanes, for example. In this case, themounting location must be chosen to provide a good view of the blindspot along the side of the vehicle in order to pick up vehicles whichare about to pass vehicle 710. Each of the locations 734, 736 and 730provide sufficient field of view for this application although the spaceimmediately adjacent to the vehicle could be missed. Alternate locationsinclude mounting onto the outside rear view mirror or the addition of aunit in the rear window or C-Pillar, in which case, the contents ofareas other than the side of the vehicle would be monitored. Usingseveral receivers in various locations as disclosed above would providefor a monitoring system which monitors all of the areas around thevehicle. The mirror location, however, does leave the device vulnerableto being covered with ice, snow and dirt.

[0221] In many cases, neural networks are used to identify objectsexterior of the vehicle and then an icon is displayed on a heads-updisplay, for example, which provides core control over the brightness ofthe image and permits the driver to more easily recognize the object.

[0222] In both cases of the anticipatory sensor and blind spot detector,the infrared transmitter and CCD array system provides mainly imageinformation to permit recognition of the object in the vicinity ofvehicle 710, whether the object is alongside the vehicle, in a blindspot of the driver, in front of the vehicle or behind the vehicle, theposition of the object being detected being dependent on the positionand orientation of the receiver(s). To complete the process, distanceinformation is also require as well as velocity information, which canin general be obtained by differentiating the position data. This can beaccomplished by any one of the several methods discussed above, such aswith a pulsed laser radar system, as well as with a radar system.

[0223] Radar systems, which may not be acceptable for use in theinterior of the vehicle, are now commonly used in sensing applicationsexterior to the vehicle, police radar being one well-known example.Miniature radar systems are now available which are inexpensive and fitwithin the available space. Such systems are disclosed in the McEwanpatents described above. Another advantage of radar in this applicationis that it is easy to get a transmitter with a desirable divergenceangle so that the device does not have to be aimed. One particularlyadvantageous mode of practicing the invention for these cases,therefore, is to use radar and a second advantageous mode is the pulsedlaser radar system, along with a CCD array, although the use of two CCDarrays or the acoustical systems are also good choices. The acousticalsystem has the disadvantage of being slower than the laser radar deviceand must be mounted outside of the vehicle where it may be affected bythe accumulation of deposits onto the active surface.

[0224] In a preferred implementation, transmitter 730 is an infraredtransmitter and receivers 734, 736 and 738 are CCD transducers thatreceive the reflected infrared waves from vehicle 720. In theimplementation shown in FIG. 7, an exterior airbag 790 is shown whichdeploys in the event that a side impact is about to occur as describedin U.S. patent application Ser. No. 08/247,760.

[0225] Referring now to FIG. 15, a schematic of the use of one or morereceivers 734, 736, 738 to affect another system in the vehicle isshown. The general exterior monitoring system, or blind spot monitoringsystem if the environment exterior of the vehicle is not viewable by thedriver in the normal course of driving the vehicle, includes one or morereceivers 734,736,738 positioned at various locations on the vehicle forthe purpose of receiving waves from the exterior environment. Instead ofwaves, and to the extent different than waves, the receivers 734, 736,738 could be designed to receiver energy or radiation.

[0226] The waves received by receivers 734, 736, 738 contain informationabout the exterior objects in the environment, such waves either havingbeen generated by or emanating from the exterior objects or reflectedfrom the exterior objects such as is the case when the optionaltransmitter 730 is used. The electronic module/processor 740 containsthe necessary circuitry 742,744 and trained pattern recognition means(neural computer 745) to drive the transmitter 730 when present andprocess the received waves to provide a classification, identificationand/or location of the exterior object. The classification,identification and/or location is then used to show an image or icon ona display 748 viewable to the driver. Also, the classification,identification and/or location of the objects could be used for airbagcontrol, i.e., control of the deployment of the exterior airbag 790 (orany other airbags for that matter), for the control of the headlightdimmers (as discussed below with reference to FIG. 8), or in general,for any other system whose operation might be changed based on thepresence of exterior objects.

[0227]FIG. 8 illustrates the exterior monitoring system for use indetecting the headlights of an oncoming vehicle or the taillights of avehicle in front of vehicle 810. In this embodiment, the CCD array isdesigned to be sensitive to visible light and a separate source ofillumination is not used. Once again for some applications, the key tothis technology is the use of trained pattern recognition algorithms andparticularly the artificial neural network. Here, as in the other casesabove and in the patents and patent applications referenced above, thepattern recognition system is trained to recognize the pattern of theheadlights of an oncoming vehicle or the tail lights of a vehicle infront of vehicle 810 and to then dim the headlights when either of theseconditions is sensed. It is also trained to not dim the lights for otherreflections such as reflections off of a sign post or the roadway. Oneproblem is to differentiate taillights where dimming is desired fromdistant headlights where dimming is not desired. Three techniques areused: (i) measurement of the spacing of the light sources, (ii)determination of the location of the light sources relative to thevehicle, and (iii) use of a red filter where the brightness of the lightsource through the filter is compared with the brightness of theunfiltered light. In the case of the taillight, the brightness of thered filtered and unfiltered light is nearly the same while there is asignificant difference for the headlight case. In this situation, eithertwo CCD arrays are used, one with a filter, or a filter which can beremoved either electrically, such as with a liquid crystal, ormechanically.

[0228] The headlights of oncoming vehicles frequently make it difficultfor the driver of a vehicle to see the road and safely operate thevehicle. This is a significant cause of accidents and much discomfort.The problem is especially severe during bad weather where rain can causemultiple reflections. Visors are now used to partially solve thisproblem but they do so by completely blocking the view through a largeportion of the window and therefore cannot be used to cover the entirewindshield. Similar problems happen when the sun is setting or risingand the driver is operating the vehicle in the direction of the sun. Thevehicle interior monitoring system of this invention can contribute tothe solution of this problem by determining the position of the driver'seyes as discussed above. If separate sensors are used to sense thedirection of the light from the on-coming vehicle or the sun and throughthe use of electrochromic glass or a liquid crystal assembly, a portionof the windshield can be darkened to impose a filter between the eyes ofthe driver and the light source. Electrochromic glass is a materialwhere the color of the glass can be changed through the application ofan electric current. By dividing the windshield into a controlled gridor matrix of contiguous areas and through feeding the current into thewindshield from orthogonal directions, selective portions of thewindshield can be darkened as desired using either the electrocbromic,liquid crystal or a similar technology. There are other technologiescurrently under development that perform in a similar manner as liquidcrystals. The term “liquid crystal” as used herein, therefore, will beused to represent the class of all such materials where the opticaltransmissibility can be varied electrically or electronically.Electrochromic products are available from Gentex of Zeeland, Mich., andDonnelly of Holland, Mich.

[0229]FIG. 9 illustrates how such a system operates. A sensor 910located on vehicle 902 determines the direction of the light from theheadlights of oncoming vehicle 904. Sensor 910 is comprised of a lensand a CCD array with appropriate electronic circuitry which determineswhich elements of the CCD array are being most brightly illuminated. Analgorithm stored in control circuitry 120 then calculates the directionof the light from the oncoming headlights based on the information fromthe CCD array. Transducers 110, 111, 113 and 114 determine the probablelocation of the eyes of the operator 101 of vehicle 902 in a manner suchas described above. In this case, however, the determination of theprobable locus of the driver's eyes is made with an accuracy of adiameter for each eye of about 3 inches (7.5 cm). This calculationsometimes will be in error and provision is made for the driver to makean adjustment to correct for this error as described below.

[0230] The windshield 916 of vehicle 902 comprises a liquid crystal, orsimilar technology, and is selectively darkened at area 918 due to theapplication of a current along perpendicular directions 922 and 924 ofwindshield 916 (See FIG. 9A). The particular portion of the windshieldto be darkened is determined by control circuitry 120. Once thedirection of the light from the oncoming vehicle is known and thelocations of the driver's eyes are known, it is a matter of simpletrigonometry to determine which areas of the windshield matrix should bedarkened to impose a filter between the headlights and the driver'seyes. This is accomplished by control circuitry 120. A separate controlsystem, not shown, located on the instrument panel, or at some otherconvenient location, allows the driver to select the amount of darkeningaccomplished by the system from no darkening to maximum darkening. Inthis manner, the driver can select the amount of light which is filteredto suit his particular physiology. The sensor 910 can either be designedto respond to a single light source or to multiple light sources to besensed and thus multiple portions of the vehicle windshield to bedarkened.

[0231] As mentioned above, the calculations of the location of thedriver's eyes may be in error and therefore provision can be made tocorrect for this error. One such system permits the driver to adjust thecenter of the darkened portion of the windshield to correct for sucherrors through a knob on the instrument panel, steering wheel or otherconvenient location. Another solution permits the driver to make theadjustment by slightly moving his head. Once a calculation as to thelocation of the driver's eyes has been made, that calculation is notchanged even though the driver moves his head slightly. It is assumedthat the driver will only move his head to center the darkened portionof the windshield to optimally filter the light from the oncomingvehicle. The monitoring system will detect this initial head motion andmake the correction automatically for future calculations.

[0232] In the applications discussed and illustrated above, the sourceand receiver of the electromagnetic radiation have been mounted in thesame package. This is not necessary and in some implementations, theillumination source will be mounted elsewhere. For example, a laser beamcan be used which is directed along an axis which bisects the anglebetween the center of the seat volume and two of the arrays. Such a beammay come from the A-Pillar, for example. The beam, which may besupplemental to the main illumination system, provides a pointreflection from the occupying item that, in most cases, can been seen bytwo receivers. Triangulation thereafter can precisely determination thelocation of the illuminated point. This point can be moved to providemore information. In another case where it is desired to track the headof the occupant, for example, several such beams can be directed at theoccupant's head during pre-crash braking or even during a crash toprovide the fastest information as to the location of the head of theoccupant for the fastest tracking of the motion of the occupant's head.Since only a few pixels are involved, even the calculation time isminimized.

[0233] In most of the applications above the assumption has been madethat either a uniform field of light or a scanning spot of light will beprovided. This need not be the case. The light that is emitted ortransmitted to illuminate the object can be structured light. Structuredlight can take many forms starting with, for example, a rectangular orother macroscopic pattern of light and dark can be superimposed on thelight by passing it through a filter. If a similar pattern is interposedbetween the reflections and the camera, a sort of pseudo-interferencepattern can result sometimes known as Moire patterns. A similar effectcan be achieved by polarizing transmitted light so that different partsof the object that is being illuminated are illuminated with light ofdifferent polarization. Once again by viewing the reflections through asimilarly polarized array, information can be obtained as to where thesource of light came from which is illuminating a particular object.Thus, any of the transmitter/receiver assemblies or transducers in anyof the embodiments above using optics can be designed to use structuredlight.

[0234] One consideration when using structured light is that the sourceof structured light cannot be exactly co-located with the array becausein this case, the pattern projected will not change as a function of thedistance between the array and the object and thus the distance betweenthe array and the object cannot be determined. Thus, it is usuallynecessary to provide a displacement between the array and the lightsource. For example, the light source can surround the array, be on topof the array or on one side of the array. The light source can also havea different virtual source, i.e., it can appear to come from behind ofthe array or in front of the array.

[0235] The goal is to determine the direction that a particular ray oflight had when it was transmitted from the source. Then by knowing whichpixels were illuminated by the reflected light ray along with thegeometry of the vehicle, the distance to the point of reflection off ofthe object can be determined. This is particularly effective if thelight source is not collocated with the CCD array. If a particular lightray, for example, illuminates an object surface which is near to thesource then the reflection off of that surface will illuminate a pixelat a particular point on the CCD array. If the reflection of the sameray however occurs from a more distant surface, then a different pixelwill be illuminated in the CCD array. In this manner the distance fromthe surface of the object to the CCD can be determined by triangulationformulas. Similarly if a given pixel is illuminated in the CCD from areflection of a particular ray of light from the transmitter, and if weknow the direction that that ray of light was sent from the transmitter,then the distance to the object at the point of reflection can bedetermined. If each ray of light is individually recognizable andtherefore can be correlated to the angle at which it was transmitted,then a full three-dimensional image can be obtained of the object thatsimplifies the identification problem.

[0236] The coding of the light rays coming from the transmitter can beaccomplished in many ways. One method is to polarize the light bypassing the light through a filter whereby the polarization is acombination of the amount and angle of the polarization. This gives twodimensions that can therefore be used to fix the angle that the lightwas sent. Another method is to superimpose an analog or digital signalonto the light which could be done, for example, by using an addressablelight valve, such as. a liquid crystal filter, electrochromic filter,or, preferably, a garnet crystal array. Each pixel in this array wouldbe coded such that it could be identified at the CCD.

[0237] The technique described above is dependent upon either changingthe polarization or using the time domain to identify particulartransmission angles with particular reflections. Spatial patterns canalso be imposed on the transmitted light which generally goes under theheading of structured light. The concept is that if a pattern isidentifiable then either the direction of transmitted light can bedetermined or, if the transmission source is collocated with thereceiver, then the pattern expands as it travels toward the object andthen, by determining the size of the received pattern, the distance tothe object can be determined. In some cases Moier pattern techniques areutilized.

[0238] When the illumination source is not placed at the same locationas the receiving array, it is typically placed at an angle such as 45degrees. At least two other techniques can be considered. One is toplace the illumination source at 90 degrees to the CCD array. In thiscase only those surface elements that are closer to the receiving arraythen previous surfaces are illuminated. Thus significant information canbe obtained as to the profile of the object. In fact, if no object isoccupying the seat, then there will be no reflections except from theseat itself. This provides a very powerful technique for determiningwhether the seat is occupied and where the initial surfaces of theoccupying item are located.

[0239] The particular radiation field of the transmitting transducer canalso be important to some implementations of this invention. In sometechniques the object which is occupying the seat is the only part ofthe vehicle which is illuminated. Extreme care is exercised in shapingthe field of light such that this is true. For example, the objects areilluminated in such a way that reflections from the door panel do notoccur. Ideally if only the items which occupy the seat can beilluminated then the problem of separating the occupant from theinterior vehicle passenger compartment surfaces can be more easilyaccomplished.

[0240] Another variant on the invention is to use no illumination sourceat all. In this case, the entire visible and infrared spectrum will beused. CMOS arrays are now available with very good night visioncapabilities making it possible to see and image an occupant in very lowlight conditions.

[0241] A further consideration to this invention is to use the motion ofthe occupant, as determined from successive differential arrays, forexample, to help identify that there is in fact a living objectoccupying the seat, or for other purposes.

[0242]FIG. 10 shows a schematic illustration of a system for controllingoperation of a vehicle based on recognition of an authorized individualin accordance with the invention. One or more images of the passengercompartment 500 are received at 502 and data derived therefrom at 504.Multiple image receivers may be provided at different locations. Thedata derivation may entail any one or more of numerous types of imageprocessing techniques such as those described in U.S. patent applicationSer. No. 09/474,147 incorporated by reference herein, including thosedesigned to improve the clarity of the image. A pattern recognitionalgorithm, e.g., a neural network, is trained in a training phase 506 torecognize authorized individuals. The training phase can be conductedupon purchase of the vehicle by the dealer or by the owner afterperforming certain procedures provided to the owner, e.g., entry of asecurity code or key. In the training phase for a theft preventionsystem, the authorized driver(s) would sit themselves in the passengerseat and optical images would be taken and processed to obtain thepattern recognition algorithm. A processor 508 is embodied with thepattern recognition algorithm thus trained to identify whether a personis the individual by analysis of subsequently obtained data derived fromoptical images 504. The pattern recognition algorithm in processor 508outputs an indication of whether the person in the image is anauthorized individual for which the system is trained to identify. Asecurity system 510 enable operations of the vehicle when the patternrecognition algorithm provides an indication that the person is anindividual authorized to operate the vehicle and prevents operation ofthe vehicle when the pattern recognition algorithm does not provide anindication that the person is an individual authorized to operate thevehicle.

[0243] Optionally, an optical transmitting unit 512 is provided totransmit electromagnetic energy into the passenger compartment such thatelectromagnetic energy transmitted by the optical transmitting unit isreflected by the person and received by the optical image receptiondevice 502.

[0244] As noted above, several different types of optical receptiondevices can be used including a CCD array, a CMOS array, any type oftwo-dimensional image receiver, any type of three-dimensional imagereceiver, an active pixel camera and an HDRC camera.

[0245] The processor 508 can also be trained to determine the positionof the individuals included in the images obtained by the optical imagereception device, as well as the distance between the optical imagereception devices and the individuals.

[0246] Instead of a security system, another component in the vehiclecan be affected or controlled based on the recognition of a particularindividual. For example, the rear view mirror, seat, seat belt anchoragepoint, headrest, pedals, steering wheel, entertainment system,air-conditioning/ventilation system can be adjusted.

[0247]FIG. 11 is a schematic illustration of a method for controllingoperation of a vehicle based on recognition of a person as one of a setof authorized individuals. Although the method is described and shownfor permitting or preventing ignition of the vehicle based onrecognition of an authorized driver, it can be used to control for anyvehicle component, system or subsystem based on recognition of anindividual.

[0248] Initially, the system is set in a training phase 514 in whichimages including the authorized individuals are obtained by means of atleast one optical receiving unit 516 and a pattern recognition algorithmis trained based thereon 518, usually after application of one or moreimage processing techniques to the images. The authorized individual(s)occupy the passenger compartment and have their picture taken by theoptical receiving unit to enable the formation of a database on whichthe pattern recognition algorithm is trained. Training can be performedby any known method in the art.

[0249] The system is then set in an operational phase 520 wherein animage is obtained 522, including the driver when the system is used fora security system. If the system is used for component adjustment, thenthe image would include any passengers or other occupying items in thevehicle. The obtained image, or images if multiple optical receivingunits are used, are input into the pattern recognition algorithm 524,preferably after some image processing, and a determination is madewhether the pattern recognition algorithm indicates that the imageincludes an authorized driver 526. If so, ignition of the vehicle isenabled 528, or the vehicle may actually be started automatically. Ifnot, an alarm is sounded and/or the police may be contacted 530.

[0250] The image processing techniques may be, when each image isconstituted by an array of pixels, elimination of pixels from the imageswhich are present in multiple images and/or comparing the images withstored arrays of pixels and eliminating pixels from the images which arepresent in the stored arrays of pixels.

[0251]FIG. 12 shows the components of the manner in which an environmentof the vehicle, designated 600, is monitored. The environment may eitherbe an interior environment, the entire passenger compartment or only apart thereof, or an exterior environment. An active pixel camera 602obtains images of the environment and provides the images or arepresentation thereof, or data derived from, to a processor 604. Theprocessor 604 determines at least one characteristic of an object in theenvironment based on the images obtained by the active pixel camera 602,e.g., the presence of an object in the environment, the type of objectin the environment, the position of an object in the environment and thevelocity of an object in the environment. Several active pixel camerascan be provided, each focusing on a different area of the environment,although some overlap is desired. Instead of an active pixel camera orarray, a single light-receiving pixel can be used.

[0252] An active pixel camera can be arranged in various locations inthe vehicle including in a headliner, roof, ceiling, an A-pillar, aB-pillar and a C-pillar. Images of the front seat area or the rear seatarea can be obtained by proper placement and orientation of the cameras.

[0253] Once a characteristic of the object is obtained, it can be usedfor numerous purposes. For example, the processor can be programmed tocontrol a reactive component, system or subsystem 606 based on thedetermined characteristic of the object. When the reactive component isan airbag assembly including one or more airbags, the processor cancontrol one or more deployment parameters of the airbag(s).

[0254] As to the manner in which the apparatus operates, reference ismade to FIG. 13, wherein as a first step 610, one or more images of theenvironment are obtained. One or more characteristics of objects in theimages are determined at 612, using, for example, pattern recognitiontechniques, and then one or more components are controlled at 614 basedon the determined characteristics. The process of obtaining andprocessing the images, or the processing of data derived from the imagesor data representative of the images, is periodically continued at leastthroughout the operation of the vehicle.

[0255]FIG. 14 shows the components for measuring the position of anobject in an environment of or about the vehicle. A light source 616directs modulated light into the environment and at least onelight-receiving pixel or an array of pixels 618 receives the modulatedlight after reflection by any objects in the environment. A processor620 determines the distance between any objects from which the modulatedlight is reflected and the light source based on the reception of themodulated light by the pixel(s) 618. To provide the modulated light, adevice or component for modulating a frequency of the light 622 areprovided. Also, a device for providing a correlation pattern in a formof code division modulation of the light can be used. The pixel may be aphoto diode such as a PIN or avalanche diode.

[0256] The processor 620 includes appropriate circuitry to determine thedistance between any objects from which any pulse of light is reflectedand the light source 616. For example, the processor 620 can determinethis distance based on a difference in time between the emission of apulse of light by the light source 616 and the reception of light by thepixel 618.

[0257] Thus, one method described above for determining theidentification and position of objects in a passenger compartment of avehicle in accordance with the invention comprises the steps oftransmitting electromagnetic waves (optical or non-optical) into thepassenger compartment from one or more locations, obtaining a pluralityof images of the interior of the passenger compartment from severallocations, and comparing the images of the interior of the passengercompartment with stored images representing different arrangements ofobjects in the passenger compartment to determine which of the storedimages match most closely to the images of the interior of the passengercompartment such that the identification of the objects and theirposition is obtained based on data associated with the stored images.The electromagnetic waves may be transmitted from transmitter/receiverassemblies positioned at different locations around a seat such thateach assembly is situated in a middle of a side of the ceilingsurrounding the seat or in the middle of the headliner directly abovethe seat. The method would thus be operative to determine theidentification and/or position of the occupants of that seat. Eachassembly may comprise an optical transmitter (such as an infrared LED,an infrared LED with a diverging lens, a laser with a diverging lens anda scanning laser assembly) and an optical array (such as a CCD array anda CMOS array). The optical array is thus arranged to obtain the imagesof the interior of the passenger compartment represented by a matrix ofpixels. To enhance the method, prior to the comparison of the images,each obtained image or output from each array may be compared with aseries of stored images or arrays representing different unoccupiedstates of the passenger compartment, such as different positions of theseat when unoccupied, and each stored image or array is subtracted fromthe obtained image or acquired array. Another way to determine whichstored image matches most closely to the images of the interior of thepassenger compartment is to analyze the total number of pixels of theimage reduced below a threshold level, and analyze the minimum number ofremaining detached pixels. Preferably, a library of stored images isgenerated by positioning an object on the seat, transmittingelectromagnetic waves into the passenger compartment from one or morelocations, obtaining images of the interior of the passengercompartment, each from a respective location, associating the imageswith the identification and position of the object, and repeating thepositioning step, transmitting step, image obtaining step andassociating step for the same object in different positions and fordifferent objects in different positions. If the objects include asteering wheel, a seat and a headrest, the angle of the steering wheel,the telescoping position of the steering wheel, the angle of the back ofthe seat, the position of the headrest and the position of the seat maybe obtained by the image comparison. One advantage of thisimplementation is that after the identification and position of theobjects are obtained, one or more systems in the vehicle, such as anoccupant restraint device or system, a mirror adjustment system, a seatadjustment system, a steering wheel adjustment system, a pedaladjustment system, a headrest positioning system, a directionalmicrophone, an air-conditioning/heating system, an entertainment system,may be affected based on the obtained identification and position of atleast one of the objects. The image comparison may entail inputting theimages or a form thereof into a neural network which provides for eachimage of the interior of the passenger compartment, an index of a storedimage that most closely matches the image of the interior of thepassenger compartment. The index is thus utilized to locate storedinformation from the matched image including, inter alia, a locus of apoint representative of the position of the chest of the person, a locusof a point representative of the position of the head of the person, oneor both ears of the person, one or both eyes of the person and the mouthof the person. Moreover, the position of the person relative to at leastone airbag or other occupant restraint system of the vehicle may bedetermined so that deployment of the airbag(s) or occupant restraintsystem is controlled based on the determined position of the person. Itis also possible to obtain information about the location of the eyes ofthe person from the image comparison and adjust the position of one ormore of the rear view mirrors based on the location of the eyes of theperson. Also, the location of the eyes of the person may be obtainedsuch that an external light source may be filtered by darkening thewindshield of the vehicle at selective locations based on the locationof the eyes of the person. Further, the location of the ears of theperson may be obtained such that a noise cancellation system in thevehicle is operated based on the location the ears of the person. Thelocation of the mouth of the person may be used to direct a directionalmicrophone in the vehicle. In addition, the location of the locus of apoint representative of the position of the chest or head (e.g., theprobable center of the chest or head) over time may be monitored by theimage comparison and one or more systems in the vehicle controlled basedon changes in the location of the locus of the center of the chest orhead over time. This monitoring may entail subtracting a most recentlyobtained image from an immediately preceding image and analyzing aleading edge of changes in the images or deriving a correlation functionwhich correlates the images with the chest or head in an initialposition with the most recently obtained images. In one particularlyadvantageous embodiment, the weight applied onto the seat is measuredand one or more systems in the vehicle are affected (controlled) basedon the measured weight applied onto the seat and the identification andposition of the objects in the passenger compartment.

[0258] In another method disclosed above for determining theidentification and position of objects in a passenger compartment of avehicle in accordance with the invention, electromagnetic waves aretransmitted into the passenger compartment from one or more locations, aplurality of images of the interior of the passenger compartment areobtained, each from a respective location, a three-dimensional map ofthe interior of the passenger compartment is created from the images,and a pattern recognition technique is applied to the map in order todetermine the identification and position of the objects in thepassenger compartment. The pattern recognition technique may be a neuralnetwork, fuzzy logic or an optical correlator or combinations thereofThe map may be obtained by utilizing a scanning laser radar system wherethe laser is operated in a pulse mode and determining the distance fromthe object being illuminated using range gating. (See, for example, H.Kage, W. Freemen, Y Miyke, E. Funstsu, K. Tanaka, K. Kyuma “Artificialretina chips as on-chip image processors and gesture-orientedinterfaces”, Optical Engineering, December, 1999, Vol. 38, Number 12,ISSN 0091-3286)

[0259] In a method disclosed above for tracking motion of a vehicularoccupant's head or chest in accordance with the invention,electromagnetic waves are transmitted toward the occupant from at leastone location, a first image of the interior of the passenger compartmentis obtained from each location, the first image being represented by amatrix of pixels, and electromagnetic waves are transmitted toward theoccupant from the same location(s) at a subsequent time and anadditional image of the interior of the passenger compartment isobtained from each location, the additional image being represented by amatrix of pixels. The additional image is subtracted from the firstimage to determine which pixels have changed in value. A leading edge ofthe changed pixels and a width of a field of the changed pixels isdetermined to thereby determine movement of the occupant from the timebetween which the first and additional images were taken. The firstimage is replaced by the additional image and the steps of obtaining anadditional image and subtracting the additional image from the firstimage are repeated such that progressive motion of the occupant isattained.

[0260] A method disclosed above for controlling deployment of anoccupant restraint system in a vehicle comprises the steps oftransmitting electromagnetic waves toward an occupant seated in apassenger compartment of the vehicle from one or more locations,obtaining a plurality of images of the interior of the passengercompartment, each from a respective location, analyzing the images todetermine the distance between the occupant and the occupant restraintsystem, and controlling deployment of the occupant restraint systembased on the determined distance between the occupant and the occupantrestraint system. The images may be analyzed by comparing the images ofthe interior of the passenger compartment with stored imagesrepresenting different arrangements of objects in the passengercompartment to determine which of the stored images match most closelyto the images of the interior of the passenger compartment, each storedimage having associated data relating to the distance between theoccupant in the image and the occupant restraint system. The imagecomparison step may entail inputting the images or a form thereof into aneural network which provides for each image of the interior of thepassenger compartment, an index of a stored image that most closelymatches the image of the interior of the passenger compartment. In aparticularly advantageous embodiment, the weight of the occupant on aseat is measured and deployment of the occupant restraint system iscontrolled based on the determined distance between the occupant and theoccupant restraint system and the measured weight of the occupant.

[0261] In another method disclosed above for determining theidentification and position of objects in a passenger compartment of avehicle, a plurality of images of the interior of the passengercompartment, each from a respective location and of radiation emanatingfrom the objects in the passenger compartment, and the images of theradiation emanating from the objects in the passenger compartment arecompared with stored images of radiation emanating from differentarrangements of objects in the passenger compartment to determine whichof the stored images match most closely to the images of the interior ofthe passenger compartment such that the identification of the objectsand their position is obtained based on data associated with the storedimages. In this embodiment, there is no illumination of the passengercompartment with electromagnetic waves. Nevertheless, the same processesdescribed above may be applied in conjunction with this method, e.g.,affecting another system based on the position and identification of theobjects, a library of stored images generated, external light sourcefiltering, noise filtering, occupant restraint system deployment controland the utilization of weight for occupant restraint system control.

[0262] Thus, disclosed above is a system to identify, locate and monitoroccupants, including their parts, and other objects in the passengercompartment and objects outside of a motor vehicle, such as anautomobile or truck, by illuminating the contents of the vehicle and/orobjects outside of the vehicle with electromagnetic radiation, andspecifically infrared radiation, or using radiation naturally emanatingfrom the object, and using one or more lenses to focus images of thecontents onto one or more arrays of charge coupled devices (CCD's) orCMOS arrays. Outputs from the CCD or CMOS arrays are analyzed byappropriate computational means employing trained pattern recognitiontechnologies, to classify, identify or locate the contents and/orexternal objects. In general, the information obtained by theidentification and monitoring system may be used to affect the operationof at least one other system in the vehicle.

[0263] When the vehicle interior monitoring system in accordance withsome embodiments of this invention is installed in the passengercompartment of an automotive vehicle equipped with a passengerprotective device, such as an inflatable airbag, and the vehicle issubjected to a crash of sufficient severity that the crash sensor hasdetermined that the protective device is to be deployed, the systemdetermines the position of the vehicle occupant relative to the airbagand disables deployment of the airbag if the occupant is positioned sothat he/she is likely to be injured by the deployment of the airbag. Inthe alternative, the parameters of the deployment of the airbag can betailored to the position of the occupant relative to the airbag, e.g., adepowered deployment.

[0264] In some implementations of the invention, several CCD or CMOSarrays are placed in such a manner that the distance from, and themotion of the occupant toward, the airbag can be monitored as atransverse motion across the field of the array. In this manner, theneed to measure the distance from the array to the object is obviated.In other implementations, the source of infrared light is a pulsemodulated laser which permits an accurate measurement of the distance tothe point of reflection through the technique of range gating to measurethe time of flight of the radiation pulse.

[0265] In some applications, a trained pattern recognition system, suchas a neural network, sensor fusion or neural-fuzzy system, is used toidentify the occupancy of the vehicle or an object exterior to thevehicle. In some of these cases, the pattern recognition systemdetermines which of a library of images most closely matches the seatedstate of a particular vehicle seat and thereby the location of certainparts of an occupant can be accurately estimated from the matchedimages, thus removing the requirement for the pattern recognition systemto locate the head of an occupant, for example.

[0266] Also disclosed above is an arrangement for determining vehicleoccupant position relative to a fixed structure within the vehicle whichcomprises an array structured and arranged to receive an image of aportion of the passenger compartment of the vehicle in which theoccupant is likely to be situated, a lens arranged between the array andthe portion of the passenger compartment, adjustment means for changingthe image received by the array, and processor means coupled to thearray and the adjustment means. The processor means determine, uponchanging by the adjustment means of the image received by the array,when the image is clearest whereby a distance between the occupant andthe fixed structure is obtainable based on the determination by theprocessor means when the image is clearest. The image may be changed byadjusting the lens, e.g., adjusting the focal length of the lens and/orthe position of the lens relative to the array, by adjusting the array,e.g., the position of the array relative to the lens, and/or by usingsoftware to perform a focusing process. The array may be arranged inseveral advantageous locations on the vehicle, e.g., on an A-pillar ofthe vehicle, above a top surface of an instrument panel of the vehicleand on an instrument panel of the vehicle and oriented to receive animage reflected by a windshield of the vehicle. The array may be a CCDarray with an optional liquid crystal or electrochromic glass filtercoupled to the array for filtering the image of the portion of thepassenger compartment. The array could also be a CMOS array. In apreferred embodiment, the processor means are coupled to an occupantprotection device and control the occupant protection device based onthe distance between the occupant and the fixed structure. For example,the occupant protection device could be an airbag whereby deployment ofthe airbag is controlled by the processor means. The processor means maybe any type of data processing unit such as a microprocessor. Thisarrangement could be adapted for determining distance between thevehicle and exterior objects, in particular, objects in a blind spot ofthe driver. In this case, such an arrangement would comprise an arraystructured and arranged to receive an image of an exterior environmentsurrounding the vehicle containing at least one object, a lens arrangedbetween the array and the exterior environment, adjustment means forchanging the image received by the array, and processor, means coupledto the array and the adjustment means. The processor means determine,upon changing by the adjustment means of the image received by thearray, when the image is clearest whereby a distance between the objectand the vehicle is obtainable based on the determination by theprocessor means when the image is clearest. As before, the image may bechanged by adjusting the lens, e.g., adjusting the focal length of thelens and/or the position of the lens relative to the array, by adjustingthe array, e.g., the position of the array relative to the lens, and/orby using software to perform a focusing process. The array may be a CCDarray with an optional liquid crystal or electrochromic glass filtercoupled to the array for filtering the image of the portion of thepassenger compartment. The array could also be a CMOS array. In apreferred embodiment, the processor means are coupled to an occupantprotection device and control the occupant protection device based onthe distance between the occupant and the fixed structure. For example,the occupant protection device could be an airbag whereby deployment ofthe airbag is controlled by the processor means. The processor means maybe any type of data processing unit such as a microprocessor.

[0267] Also, at least one of the above-listed objects is achieved by anarrangement for determining vehicle occupant presence, type and/orposition relative to a fixed structure within the vehicle, the vehiclehaving a front seat and an A-pillar. The arrangement comprises a firstarray mounted on the A-pillar of the vehicle and arranged to receive animage of a portion of the passenger compartment in which the occupant islikely to be situated, and processor means coupled to the first arrayfor determining the presence, type and/or position of the vehicleoccupant based on the image of the portion of the passenger compartmentreceived by the first array. The processor means preferably are arrangedto utilize a pattern recognition technique, e.g., a trained neuralnetwork, sensor fusion, fuzzy logic. The processor means can determinethe vehicle occupant presence, type and/or position based on the imageof the portion of the passenger compartment received by the first array.In some embodiments, a second array is arranged to receive an image ofat least a part of the same portion of the passenger compartment as thefirst array. The processor means are coupled to the second array anddetermine the vehicle occupant presence, type and/or position based onthe images of the portion of the passenger compartment received by thefirst and second arrays. The second array may be arranged at a centralportion of a headliner of the vehicle between sides of the vehicle. Thedetermination of the occupant presence, type and/or position can be usedin conjunction with a reactive component, system or subsystem so thatthe processor means control the reactive component, system or subsystembased on the determination of the occupant presence, type and/orposition. For example, if the reactive component, system or subsystem isan airbag assembly including at least one airbag, the processor meanscontrol one or more deployment parameters of the airbag(s). The arraysmay be CCD arrays with an optional liquid crystal or electrochromicglass filter coupled to the array for filtering the image of the portionof the passenger compartment. The arrays could also be CMOS arrays,active pixel cameras and HDRC cameras.

[0268] Another embodiment disclosed above is an arrangement forobtaining information about a vehicle occupant within the vehicle whichcomprises transmission means for transmitting a structured pattern oflight, e.g., polarized light, into a portion of the passengercompartment in which the occupant is likely to be situated, an arrayarranged to receive an image of the portion of the passengercompartment, and processor means coupled to the array for analyzing theimage of the portion of the passenger compartment to obtain informationabout the occupant. The transmission means and array are proximate oneanother and the information obtained about the occupant is a distancefrom the location of the transmission means and the array. The processormeans obtain the information about the occupant utilizing a patternrecognition technique. The information about of the occupant can be usedin conjunction with a reactive component, system or subsystem so thatthe processor means control the reactive component, system or subsystembased on the determination of the occupant presence, type and/orposition. For example, if the reactive component, system or subsystem isan airbag assembly including at least one airbag, the processor meanscontrol one or more deployment parameters of the airbag(s).

[0269] Another invention disclosed above is a system for controllingoperation of a vehicle based on recognition of an authorized individualcomprises a processor embodying a pattern recognition algorithm, asdefined above, trained to identify whether a person is the individual byanalyzing data derived from images and one or more optical receivingunits for receiving an optical image including the person and derivingdata from the image. Each optical receiving unit is coupled to theprocessor to provide the data to the pattern recognition algorithm tothereby obtain an indication from the pattern recognition algorithmwhether the person is the individual. A security system is arranged toenable operation of the vehicle when the pattern recognition algorithmprovides an indication that the person is an individual authorized tooperate the vehicle and prevent operation of the vehicle when thepattern recognition algorithm does not provide an indication that theperson is an individual authorized to operate the vehicle. An optionaloptical transmitting unit is provided in the vehicle for transmittingelectromagnetic energy and is arranged relative to the optical receivingunit(s) such that electromagnetic energy transmitted by the opticaltransmitting unit is reflected by the person and received by at leastone of the optical receiving units. The optical receiving units may beselected from a group consisting of a CCD array, a CMOS array, an activepixel camera and an HDRC camera. Other types of two or three-dimensionalcameras can also be used.

[0270] Instead of a security system, the individual recognition systemcan be used to control vehicular components, such as the mirrors, theseat, the anchorage point of the seatbelt, the airbag deploymentparameters including inflation rate and pressure, inflation direction,deflation rate, time of inflation, the headrest, the steering wheel, thepedals, the entertainment system and the air-conditioning/ventilationsystem. In this case, the system includes control means coupled to thecomponent for affecting the component based on the indication from thepattern recognition algorithm whether the person is the individual.

[0271] A method for controlling operation of a vehicle based onrecognition of a person as one of a set of authorized individualscomprises the steps of obtaining images including the authorizedindividuals by means of one or more optical receiving unit, derivingdata from the images, training a pattern recognition algorithm on thedata derived from the images which is capable of identifying a person asone of the individuals, then subsequently obtaining images by means ofthe optical receiving unit(s), inputting data derived from the imagessubsequently obtained by the optical receiving unit(s) into the patternrecognition algorithm to obtain an indication whether the person is oneof the set of authorized individuals, and providing a security systemwhich enables operation of the vehicle when the pattern recognitionalgorithm provides an indication that the person is one of the set ofindividuals authorized to operate the vehicle and prevents operation ofthe vehicle when the pattern recognition algorithm does not provide anindication that the person is one of the set of individuals authorizedto operate the vehicle. The data derivation from the images may entailany number of image processing techniques including eliminating pixelsfrom the images which are present in multiple images and comparing theimages with stored arrays of pixels and eliminating pixels from theimages which are present in the stored arrays of pixels. The method canalso be used to control a vehicular component based on recognition of aperson as one of a predetermined set of particular individuals. Thismethod includes the step of affecting the component based on theindication from the pattern recognition algorithm whether the person isone of the set of individuals. The components may be one or more of thefollowing: the mirrors, the seat, the anchorage point of the seatbelt,the airbag deployment parameters including inflation rate and pressure,inflation direction, deflation rate, time of inflation, the headrest,the steering wheel, the pedals, the entertainment system and theair-conditioning/ventilation system.

[0272] There has thus been shown and described, among other things, amonitoring system for monitoring both the interior and the exterior ofthe vehicle using an optical system, for example, with one or more CCDarrays or CMOS arrays, and other associated equipment which fulfills allthe objects and advantages sought after.

[0273] Lastly, it is possible to use a modulated scanning beam ofradiation and a single pixel receiver, PIN or avalanche diode, in theinventions described above. Any form of energy or radiation used abovemay be in the infrared or radar spectrums, to the extent possible, andmay be polarized and filters may be used in the receiver to block outsunlight etc. These filters may be notch filters as described above andmay be made integral with the lens as one or more coatings on the lenssurface as is well known in the art.

[0274] Also disclosed above is a vehicle including a monitoringarrangement for monitoring an environment of the vehicle which comprisesat least one active pixel camera for obtaining images of the environmentof the vehicle and a processor coupled to the active pixel camera(s) fordetermining at least one characteristic of an object in the environmentbased on the images obtained by the active pixel camera(s). The activepixel camera can be arranged in a headliner, roof or ceiling of thevehicle to obtain images of an interior environment of the vehicle, inan A-pillar or B-pillar of the vehicle to obtain images of an interiorenvironment of the vehicle, or in a roof, ceiling, B-pillar or C-pillarof the vehicle to obtain images of an interior environment of thevehicle behind a front seat of the vehicle. These mounting locations areexemplary only and not limiting.

[0275] The determined characteristic can be used to enable optimalcontrol of a reactive component, system or subsystem coupled to theprocessor. When the reactive component is an airbag assembly includingat least one airbag, the processor can be designed to control at leastone deployment parameter of the airbag(s).

[0276] Another monitoring arrangement comprises an imaging device forobtaining three-dimensional images of the environment (internal and/orexternal) and a processor embodying a pattern recognition technique forprocessing the three-dimensional images to determine at least onecharacteristic of an object in the environment based on thethree-dimensional images obtained by the imaging device. The imagingdevice can be arranged at locations throughout the vehicle as describedabove. Control of a reactive component is enabled by the determinationof the characteristic of the object.

[0277] Another arrangement for monitoring objects in or about a vehiclecomprises a generating device for generating a first signal having afirst frequency in a specific radio range, a wave transmitter arrangedto receive the signal and transmit waves toward the objects, awave-receiver arranged relative to the wave transmitter for receivingwaves transmitted by the wave transmitter after the waves haveinteracted with an object, the wave receiver being arranged to generatea second signal based on the received waves at the same frequency as thefirst signal but shifted in phase, and a detector for detecting a phasedifference between the first and second signals, whereby the phasedifference is a measure of a property of the object. The phasedifference is a measure of the distance between the object and the wavereceiver and the wave transmitter. The wave transmitter may comprise aninfrared driver and the receiver comprises an infrared diode.

[0278] A vehicle including an arrangement for measuring position of anobject in an environment of or about the vehicle comprises a lightsource capable of directing modulated light into the environment, atleast one light-receiving pixel arranged to receive the modulated lightafter reflection by any objects in the environment and a processor fordetermining the distance between any objects from which the modulatedlight is reflected and the light source based on the reception of themodulated light by the pixel(s). The pixels can constitute an array.Components for modulating a frequency of the light being directed by thelight source into the environment and for providing a correlationpattern in a form of code division modulation of the light beingdirected by the light source into the environment can be provided. Thepixel can also be a photo diode such as a PIN or avalanche diode.

[0279] Another measuring position arrangement comprises a light sourcecapable of directing individual pulses of light into the environment, atleast one array of light-receiving pixels arranged to receive lightafter reflection by any objects in the environment and a processor fordetermining the distance between any objects from which any pulse oflight is reflected and the light source based on a difference in timebetween the emission of a pulse of light by the light source and thereception of light by the array. The light source can be arranged atvarious locations in the vehicle as described above to direct light intoexternal and/or internal environments, relative to the vehicle.

[0280] Many changes, modifications, variations and other uses andapplications of the subject invention will, however, become apparent tothose skilled in the art after considering this specification and theaccompanying drawings which disclose the preferred embodiments thereof.All such changes, modifications, variations and other uses andapplications which do not depart from the spirit and scope of theinvention are deemed to be covered by the invention which is limitedonly by the following claims.

We claim:
 1. A monitoring arrangement for monitoring an environmentexterior of a vehicle, comprising: at least one receiver arranged toreceive waves from the environment exterior of the vehicle which containinformation on any objects in the environment and generate a signalcharacteristic of the received waves; and a processor coupled to said atleast one receiver and comprising pattern recognition means forprocessing the signal to provide a classification, identification orlocation of the exterior object, said pattern recognition means beingstructured and arranged to apply a pattern recognition algorithmgenerated from data of possible exterior objects and patterns ofreceived waves from the possible exterior objects; whereby a system inthe vehicle is coupled to said processor such that the operation thesystem is affected in response to the classification, identification orlocation of the exterior object.
 2. The arrangement of claim 1, whereinsaid at least one receiver comprises a pair of receivers spaced apartfrom one another.
 3. The arrangement of claim 1, wherein said at leastone receiver is arranged to receive infrared waves.
 4. The arrangementof claim 1, wherein the monitoring arrangement further comprises atransmitter for transmitting waves into the environment exterior of thevehicle whereby said at least one receiver is arranged to receive wavestransmitted by said transmitter and reflected by any exterior objects.5. The arrangement of claim 1, wherein said pattern recognition meanscomprise a neural computer.
 6. The arrangement of claim 1, wherein saidpattern recognition means comprise a neural network.
 7. The arrangementof claim 1, wherein the another system is a display viewable by thedriver and arranged to show an image or icon of the exterior object. 8.The arrangement of claim 1, wherein said at least one receiver is a CCDarray.
 9. The arrangement of claim 1, further comprising measurementmeans for measuring a distance between the exterior object and thevehicle.
 10. The arrangement of claim 9, wherein said measurement meanscomprise a radar or laser radar system.
 11. The arrangement of claim 1,wherein the another system is an airbag, deployment of said airbag beingcontrolled based on the identification of the exterior object.
 12. Thearrangement of claim 11, wherein the airbag is an externally deployedairbag.
 13. The arrangement of claim 1, wherein said processor isarranged to provide the classification of the exterior object.
 14. Thearrangement of claim 1, wherein said processor is arranged to providethe identification of the exterior object.
 15. The arrangement of claim1, wherein said processor is arranged to provide the location of theexterior object.
 16. A monitoring arrangement for monitoring anenvironment exterior of a vehicle, comprising: at least one CCD arraypositioned to obtain images of the environment exterior of the vehicle;and a processor coupled to said at least one CCD array and comprisingpattern recognition means for processing the images obtained by said atleast one CCD array to provide a classification, identification orlocation of the exterior object, whereby a system in the vehicle iscoupled to said processor such that the operation the system is affectedin response to the classification, identification or location of theexterior object.
 17. The arrangement of claim 16, wherein said patternrecognition means are structured and arranged to apply a patternrecognition algorithm generated from data of possible exterior objectsand images from said at least one CCD array of the possible exteriorobjects.
 18. The arrangement of claim 16, wherein said at least one CCDarray comprises a pair of CCD arrays spaced apart from one another. 19.The arrangement of claim 16, wherein the monitoring arrangement furthercomprises a transmitter for transmitting infrared waves into theenvironment exterior of the vehicle.
 20. The arrangement of claim 16,wherein said pattern recognition means comprise a neural computer. 21.The arrangement of claim 16, wherein said pattern recognition meanscomprise a neural network.
 22. The arrangement of claim 16, wherein theanother system is a display viewable by the driver and arranged to showan image or icon of the exterior object.
 23. The arrangement of claim16, further comprising measurement means for measuring a distancebetween the exterior object and the vehicle.
 24. The arrangement ofclaim 23, wherein said measurement means comprise a radar or laser radarsystem.
 25. The arrangement of claim 16, wherein the another system isan airbag, deployment of said airbag being controlled based on theidentification of the exterior object.
 26. The arrangement of claim 25,wherein the airbag is an externally deployed airbag.
 27. The arrangementof claim 16, wherein said processor is arranged to provide theclassification of the exterior object.
 28. The arrangement of claim 16,wherein said processor is arranged to provide the identification of theexterior object.
 29. The arrangement of claim 16, wherein said processoris arranged to provide the location of the exterior object.
 30. Avehicle including a monitoring arrangement for monitoring an environmentexterior of the vehicle, the monitoring arrangement comprising: at leastone receiver arranged on a rear view mirror of the vehicle to receivewaves from the environment exterior of the vehicle which containinformation on any objects in the environment and generate a signalcharacteristic of the received waves; and a processor coupled to said atleast one receiver and arranged to classify, identify or locate theexterior object based on the signal; whereby a system in the vehicle iscoupled to said processor such that the operation the system is affectedin response to the classification, identification or location of theexterior object.
 31. The vehicle of claim 30, wherein said processorcomprises pattern recognition means for processing the signal to providethe classification, identification or location of the exterior object,said pattern recognition means being structured and arranged to apply apattern recognition algorithm generated from data of possible exteriorobjects and patterns of received waves from the possible exteriorobjects.
 32. The vehicle of claim 30, wherein said at least one receivercomprises a pair of receivers spaced apart from one another.
 33. Thevehicle of claim 30, wherein said at least one receiver is arranged toreceive infrared waves.
 34. The vehicle -of claim 30, wherein themonitoring arrangement further comprises a transmitter for transmittingwaves into the environment exterior of the vehicle whereby said at leastone receiver is arranged to receive waves transmitted by saidtransmitter and reflected by any exterior objects.
 35. The vehicle ofclaim 30, wherein said pattern recognition means comprise a neuralcomputer.
 36. The vehicle of claim 30, wherein said pattern recognitionmeans comprise a neural network.
 37. The vehicle of claim 30, whereinsaid at least one receiver is a CCD array.
 38. The vehicle of claim 30,further comprising measurement means for measuring a distance betweenthe exterior object and the vehicle.
 39. The vehicle of claim 38,wherein said measurement means comprise a radar or laser radar system.40. A monitoring arrangement for monitoring an environment exterior of avehicle, comprising: a plurality of receivers arranged apart from oneanother and to receive waves from the environment exterior of thevehicle which contain information on any objects in the environment andgenerate a signal characteristic of the received waves; and a processorcoupled to said receivers and arranged to classify, identify or locatethe exterior object based on the signals generated by said receivers,whereby a system in the vehicle is coupled to said processor such thatthe operation of the system is affected in response to theclassification, identification or location of the exterior object. 41.The arrangement of claim 40, wherein said processor comprises patternrecognition means for processing the signal to provide theclassification, identification or location of the exterior object, saidpattern recognition means being structured and arranged to apply apattern recognition algorithm generated from data of possible exteriorobjects and patterns of received waves from the possible exteriorobjects.
 42. The arrangement of claim 40, wherein said receivers arearranged to receive infrared waves.
 43. The arrangement of claim 40,wherein the monitoring arrangement further comprises a transmitter fortransmitting waves into the environment exterior of the vehicle wherebysaid receivers are arranged to receive waves transmitted by saidtransmitter and reflected by any exterior objects.
 44. The arrangementof claim 40, wherein said pattern recognition means comprise a neuralcomputer.
 45. The arrangement of claim 40, wherein said patternrecognition means comprise a neural network.
 46. The arrangement ofclaim 40, wherein the another system is a display viewable by the driverand arranged to show an image or icon of the exterior object.
 47. Thearrangement of claim 40, wherein said receivers are CCD arrays.
 48. Thearrangement of claim 40, further comprising measurement means formeasuring a distance between the exterior object and the vehicle. 49.The arrangement of claim 48, wherein said measurement means comprise aradar or laser radar system.
 50. The arrangement of claim 40, whereinthe another system is an airbag, said output means being arranged tocontrol deployment of said airbag based on the identification of theexterior object.
 51. The arrangement of claim 50, wherein the airbag isan externally deployed airbag.
 52. A vehicle including a monitoringarrangement for monitoring an environment exterior of the vehicle, themonitoring arrangement comprising: at least one receiver arranged on aB-pillar of the vehicle to receive waves from the environment exteriorof and on the side of the vehicle which contain information on anyobjects in the environment and generate a signal characteristic of thereceived waves; and a processor coupled to said at least one receiverand arranged to classify, identify or locate the exterior object basedon the signal; whereby a system in the vehicle is coupled to saidprocessor such that the operation the system is affected in response tothe classification, identification or location of the exterior object.53. A vehicle including a monitoring arrangement for monitoring anenvironment exterior of the vehicle, the monitoring arrangementcomprising: at least one receiver arranged in a door window trim panelof the vehicle to receive waves from the environment exterior of thevehicle which contain information on any objects in the environment andgenerate a signal characteristic of the received waves; and a processorcoupled to said at least one receiver and arranged to classify, identifyor locate the exterior object based on the signal; whereby a system inthe vehicle is coupled to said processor such that the operation thesystem is affected in response to the classification, identification orlocation of the exterior object.
 54. A vehicle including a monitoringarrangement for monitoring an environment exterior of the vehicle, themonitoring arrangement comprising: at least one receiver arranged on arear view mirror of the vehicle to receive waves from the environmentexterior of the vehicle which contain information on any objects in theenvironment and generate a signal characteristic of the received waves;a processor coupled to said at least one receiver and arranged toclassify, identify or locate the exterior object based on the signal;and a display viewable by the driver; said display being coupled to saidprocessor such that the presence and classification, identification orlocation of the exterior object is displayed on said display.
 55. Avehicle including a monitoring arrangement for monitoring an environmentexterior of the vehicle, the monitoring arrangement comprising: at leastone receiver arranged to receive waves from the environment exterior ofthe vehicle which contain information on any objects in the environmentand generate a signal characteristic of the received waves; and aprocessor coupled to said at least one receiver and comprising patternrecognition means for processing the signal to provide a classification,identification or location of the exterior object, said patternrecognition means being structured and arranged to apply a patternrecognition algorithm generated from data of possible exterior objectsand patterns of received waves from the possible exterior objects;whereby a system in the vehicle is coupled to said processor such thatthe operation the system is affected in response to the classification,identification or location of the exterior object.
 56. The vehicle ofclaim 55, wherein said at least one receiver is arranged on a B-pillarof the vehicle to receive waves from a side of the vehicle.
 57. Thevehicle of claim 55, wherein said at least one receiver is arranged in adoor window trim panel.
 58. The vehicle of claim 55, wherein themonitoring arrangement further comprises a transmitter for transmittingwaves into the environment exterior of the vehicle whereby said at leastone receiver is arranged to receive waves transmitted by saidtransmitter and reflected by any exterior objects.
 59. The vehicle ofclaim 55, wherein said at least one receiver is arranged to receivewaves from a blind spot of the vehicle.
 60. The vehicle of claim 55,wherein the another system is a display viewable by the driver andarranged to show an image or icon of the exterior object.
 61. Thevehicle of claim 55, wherein said at least one receiver is mounted on arear view mirror or in a rear window.
 62. The vehicle of claim 55,wherein said at least one receiver is mounted in a C-pillar of thevehicle.
 63. The vehicle of claim 55, further comprising measurementmeans for measuring a distance between the exterior object and thevehicle.
 64. The vehicle of claim 55, wherein the another system is anairbag, deployment of said airbag being controlled based on theidentification of the exterior object.
 65. A vehicle including amonitoring arrangement for monitoring an environment exterior of thevehicle, the monitoring arrangement comprising: at least one CCD arraypositioned to obtain images of the environment exterior of the vehicle;and a processor coupled to said at least one CCD array and comprisingpattern recognition means for processing the images obtained by said atleast one CCD array to provide a classification, identification orlocation of the exterior object, whereby a system in the vehicle iscoupled to said processor such that the operation the system is affectedin response to the classification, identification or location of theexterior object.
 66. The vehicle of claim 65, wherein said patternrecognition means are structured and arranged to apply a patternrecognition algorithm generated from data of possible exterior objectsand images from said at least one CCD array of the possible exteriorobjects.
 67. The vehicle of claim 65, wherein said at least one CCDarray is arranged on a B-pillar of the vehicle to obtain images from aside of the vehicle.
 68. The vehicle of claim 65, wherein said at leastone CCD array is arranged in a door window trim panel.
 69. The vehicleof claim 65, wherein said at least one CCD array comprises a pair of CCDarrays spaced apart from one another.
 70. The vehicle of claim 65,wherein the monitoring arrangement further comprises a transmitter fortransmitting infrared waves into the environment exterior of thevehicle.
 71. The vehicle of claim 65, wherein said at least one CCDarray is arranged to obtain images from a blind spot of the vehicle. 72.The vehicle of claim 65, wherein the another system is a displayviewable by the driver and arranged to show an image or icon of theexterior object.
 73. The vehicle of claim 65, wherein said at least oneCCD array is mounted on a rear view mirror or in a rear window.
 74. Thevehicle of claim 65, wherein said at least one CCD array is mounted in aC-pillar of the vehicle.
 75. The vehicle of claim 65, further comprisingmeasurement means for measuring a distance between the exterior objectand the vehicle.
 76. The vehicle of claim 65, wherein the another systemis an airbag, deployment of said airbag being controlled based on theidentification of the exterior object.
 77. A vehicle including amonitoring arrangement for monitoring an environment exterior of thevehicle, the monitoring arrangement comprising: a plurality of receiversarranged apart from one another and to receive waves from theenvironment exterior of the vehicle which contain information on anyobjects in the environment and generate a signal characteristic of thereceived waves; and a processor coupled to said receivers and arrangedto classify, identify or locate the exterior object based on the signalsgenerated by said receivers, whereby a system in the vehicle is coupledto said processor such that the operation of the system is affected inresponse to the classification, identification or location of theexterior object.
 78. The vehicle of claim 77, wherein said processorcomprises pattern recognition means for processing the signal to providethe classification, identification or location of the exterior object,said pattern recognition means being structured and arranged to apply apattern recognition algorithm generated from data of possible exteriorobjects and patterns of received waves from the possible exteriorobjects.
 79. The vehicle of claim 77, wherein said receivers arearranged on a B-pillar of the vehicle to receive waves from a side ofthe vehicle.
 80. The vehicle of claim 77, wherein said receivers arearranged to receive infrared waves.
 81. The vehicle of claim 77, whereinthe monitoring arrangement further comprises a transmitter fortransmitting waves into the environment exterior of the vehicle wherebysaid receivers are arranged to receive waves transmitted by saidtransmitter and reflected by any exterior objects.
 82. The vehicle ofclaim 77, wherein said receivers are arranged to receive waves from ablind spot of the vehicle.
 83. The vehicle of claim 77, wherein theanother system is a display viewable by the driver and arranged to showan image or icon of the exterior object.
 84. The vehicle of claim 77,further comprising measurement means for measuring a distance betweenthe exterior object and the vehicle.
 85. The vehicle of claim 77,wherein the another system is an airbag, said output means beingarranged to control deployment of said airbag based on theidentification of the exterior object.